doc/source/guides/multinode-lab.rst

==============
Multi-Node Lab
==============

Here is OpenStack in a realistic test configuration with multiple
physical servers.

Prerequisites Linux & Network
-----------------------------

Minimal Install
~~~~~~~~~~~~~~~

You need to have a system with a fresh install of Linux. You can
download the `Minimal
CD <https://help.ubuntu.com/community/Installation/MinimalCD>`__ for
Ubuntu releases since DevStack will download & install all the
additional dependencies. The netinstall ISO is available for
`Fedora <http://mirrors.kernel.org/fedora/releases/18/Fedora/x86_64/iso/Fedora-20-x86_64-netinst.iso>`__
and
`CentOS/RHEL <http://mirrors.kernel.org/centos/6.5/isos/x86_64/CentOS-6.5-x86_64-netinstall.iso>`__.

Install a couple of packages to bootstrap configuration:

::

    apt-get install -y git sudo || yum install -y git sudo

Network Configuration
~~~~~~~~~~~~~~~~~~~~~

The first iteration of the lab uses OpenStack's FlatDHCP network
controller so only a single network will be required. It should be on
its own subnet without DHCP; the host IPs and floating IP pool(s) will
come out of this block. This example uses the following:

-  Gateway: 192.168.42.1
-  Physical nodes: 192.168.42.11-192.168.42.99
-  Floating IPs: 192.168.42.128-192.168.42.254

Configure each node with a static IP. For Ubuntu edit
``/etc/network/interfaces``:

::

    auto eth0
    iface eth0 inet static
        address 192.168.42.11
        netmask 255.255.255.0
        gateway 192.168.42.1

For Fedora and CentOS/RHEL edit
``/etc/sysconfig/network-scripts/ifcfg-eth0``:

::

    BOOTPROTO=static
    IPADDR=192.168.42.11
    NETMASK=255.255.255.0
    GATEWAY=192.168.42.1

Installation shake and bake
---------------------------

Add the DevStack User
~~~~~~~~~~~~~~~~~~~~~

OpenStack runs as a non-root user that has sudo access to root. There is
nothing special about the name, we'll use ``stack`` here. Every node
must use the same name and preferably uid. If you created a user during
the OS install you can use it and give it sudo privileges below.
Otherwise create the stack user:

::

    groupadd stack
    useradd -g stack -s /bin/bash -d /opt/stack -m stack

This user will be making many changes to your system during installation
and operation so it needs to have sudo privileges to root without a
password:

::

    echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

From here on use the ``stack`` user. **Logout** and **login** as the
``stack`` user.

Set Up Ssh
~~~~~~~~~~

Set up the stack user on each node with an ssh key for access:

::

    mkdir ~/.ssh; chmod 700 ~/.ssh
    echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyYjfgyPazTvGpd8OaAvtU2utL8W6gWC4JdRS1J95GhNNfQd657yO6s1AH5KYQWktcE6FO/xNUC2reEXSGC7ezy+sGO1kj9Limv5vrvNHvF1+wts0Cmyx61D2nQw35/Qz8BvpdJANL7VwP/cFI/p3yhvx2lsnjFE3hN8xRB2LtLUopUSVdBwACOVUmH2G+2BWMJDjVINd2DPqRIA4Zhy09KJ3O1Joabr0XpQL0yt/I9x8BVHdAx6l9U0tMg9dj5+tAjZvMAFfye3PJcYwwsfJoFxC8w/SLtqlFX7Ehw++8RtvomvuipLdmWCy+T9hIkl+gHYE4cS3OIqXH7f49jdJf jesse@spacey.local" > ~/.ssh/authorized_keys

Download DevStack
~~~~~~~~~~~~~~~~~

Grab the latest version of DevStack:

::

    git clone https://git.openstack.org/openstack-dev/devstack
    cd devstack

Up to this point all of the steps apply to each node in the cluster.
From here on there are some differences between the cluster controller
(aka 'head node') and the compute nodes.

Configure Cluster Controller
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The cluster controller runs all OpenStack services. Configure the
cluster controller's DevStack in ``local.conf``:

::

    [[local|localrc]]
    HOST_IP=192.168.42.11
    FLAT_INTERFACE=eth0
    FIXED_RANGE=10.4.128.0/20
    FIXED_NETWORK_SIZE=4096
    FLOATING_RANGE=192.168.42.128/25
    MULTI_HOST=1
    LOGFILE=/opt/stack/logs/stack.sh.log
    ADMIN_PASSWORD=labstack
    MYSQL_PASSWORD=supersecret
    RABBIT_PASSWORD=supersecrete
    SERVICE_PASSWORD=supersecrete
    SERVICE_TOKEN=xyzpdqlazydog

In the multi-node configuration the first 10 or so IPs in the private
subnet are usually reserved. Add this to ``local.sh`` to have it run
after every ``stack.sh`` run:

::

    for i in `seq 2 10`; do /opt/stack/nova/bin/nova-manage fixed reserve 10.4.128.$i; done

Fire up OpenStack:

::

    ./stack.sh

A stream of activity ensues. When complete you will see a summary of
``stack.sh``'s work, including the relevant URLs, accounts and passwords
to poke at your shiny new OpenStack. The most recent log file is
available in ``stack.sh.log``.

Configure Compute Nodes
~~~~~~~~~~~~~~~~~~~~~~~

The compute nodes only run the OpenStack worker services. For additional
machines, create a ``local.conf`` with:

::

    [[local|localrc]]
    HOST_IP=192.168.42.12 # change this per compute node
    FLAT_INTERFACE=eth0
    FIXED_RANGE=10.4.128.0/20
    FIXED_NETWORK_SIZE=4096
    FLOATING_RANGE=192.168.42.128/25
    MULTI_HOST=1
    LOGFILE=/opt/stack/logs/stack.sh.log
    ADMIN_PASSWORD=labstack
    MYSQL_PASSWORD=supersecret
    RABBIT_PASSWORD=supersecrete
    SERVICE_PASSWORD=supersecrete
    SERVICE_TOKEN=xyzpdqlazydog
    DATABASE_TYPE=mysql
    SERVICE_HOST=192.168.42.11
    MYSQL_HOST=192.168.42.11
    RABBIT_HOST=192.168.42.11
    GLANCE_HOSTPORT=192.168.42.11:9292
    ENABLED_SERVICES=n-cpu,n-net,n-api,c-sch,c-api,c-vol
    NOVA_VNC_ENABLED=True
    NOVNCPROXY_URL="http://192.168.42.11:6080/vnc_auto.html"
    VNCSERVER_LISTEN=$HOST_IP
    VNCSERVER_PROXYCLIENT_ADDRESS=$VNCSERVER_LISTEN

Fire up OpenStack:

::

    ./stack.sh

A stream of activity ensues. When complete you will see a summary of
``stack.sh``'s work, including the relevant URLs, accounts and passwords
to poke at your shiny new OpenStack. The most recent log file is
available in ``stack.sh.log``.

Cleaning Up After DevStack
~~~~~~~~~~~~~~~~~~~~~~~~~~

Shutting down OpenStack is now as simple as running the included
``unstack.sh`` script:

::

    ./unstack.sh

A more aggressive cleanup can be performed using ``clean.sh``. It
removes certain troublesome packages and attempts to leave the system in
a state where changing the database or queue manager can be reliably
performed.

::

    ./clean.sh

Sometimes running instances are not cleaned up. DevStack attempts to do
this when it runs but there are times it needs to still be done by hand:

::

    sudo rm -rf /etc/libvirt/qemu/inst*
    sudo virsh list | grep inst | awk '{print $1}' | xargs -n1 virsh destroy

Options pimp your stack
-----------------------

Additional Users
~~~~~~~~~~~~~~~~

DevStack creates two OpenStack users (``admin`` and ``demo``) and two
tenants (also ``admin`` and ``demo``). ``admin`` is exactly what it
sounds like, a privileged administrative account that is a member of
both the ``admin`` and ``demo`` tenants. ``demo`` is a normal user
account that is only a member of the ``demo`` tenant. Creating
additional OpenStack users can be done through the dashboard, sometimes
it is easier to do them in bulk from a script, especially since they get
blown away every time ``stack.sh`` runs. The following steps are ripe
for scripting:

::

    # Get admin creds
    . openrc admin admin
            
    # List existing tenants
    keystone tenant-list

    # List existing users
    keystone user-list

    # Add a user and tenant
    NAME=bob
    PASSWORD=BigSecrete
    TENANT=$NAME
    keystone tenant-create --name=$NAME
    keystone user-create --name=$NAME --pass=$PASSWORD
    keystone user-role-add --user-id=<bob-user-id> --tenant-id=<bob-tenant-id> --role-id=<member-role-id>
    # member-role-id comes from the existing member role created by stack.sh
    # keystone role-list

Swift
~~~~~

Swift requires a significant amount of resources and is disabled by
default in DevStack. The support in DevStack is geared toward a minimal
installation but can be used for testing. To implement a true multi-node
test of Swift required more than DevStack provides. Enabling it is as
simple as enabling the ``swift`` service in ``local.conf``:

::

    enable_service s-proxy s-object s-container s-account

Swift will put its data files in ``SWIFT_DATA_DIR`` (default
``/opt/stack/data/swift``). The size of the data 'partition' created
(really a loop-mounted file) is set by ``SWIFT_LOOPBACK_DISK_SIZE``. The
Swift config files are located in ``SWIFT_CONFIG_DIR`` (default
``/etc/swift``). All of these settings can be overridden in (wait for
it...) ``local.conf``.

Volumes
~~~~~~~

DevStack will automatically use an existing LVM volume group named
``stack-volumes`` to store cloud-created volumes. If ``stack-volumes``
doesn't exist, DevStack will set up a 5Gb loop-mounted file to contain
it. This obviously limits the number and size of volumes that can be
created inside OpenStack. The size can be overridden by setting
``VOLUME_BACKING_FILE_SIZE`` in ``local.conf``.

``stack-volumes`` can be pre-created on any physical volume supported by
Linux's LVM. The name of the volume group can be changed by setting
``VOLUME_GROUP`` in ``localrc``. ``stack.sh`` deletes all logical
volumes in ``VOLUME_GROUP`` that begin with ``VOLUME_NAME_PREFIX`` as
part of cleaning up from previous runs. It is recommended to not use the
root volume group as ``VOLUME_GROUP``.

The details of creating the volume group depends on the server hardware
involved but looks something like this:

::

    pvcreate /dev/sdc
    vgcreate stack-volumes /dev/sdc

Syslog
~~~~~~

DevStack is capable of using ``rsyslog`` to aggregate logging across the
cluster. It is off by default; to turn it on set ``SYSLOG=True`` in
``local.conf``. ``SYSLOG_HOST`` defaults to ``HOST_IP``; on the compute
nodes it must be set to the IP of the cluster controller to send syslog
output there. In the example above, add this to the compute node
``local.conf``:

::

    SYSLOG_HOST=192.168.42.11

Using Alternate Repositories/Branches
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The git repositories for all of the OpenStack services are defined in
``stackrc``. Since this file is a part of the DevStack package changes
to it will probably be overwritten as updates are applied. Every setting
in ``stackrc`` can be redefined in ``local.conf``.

To change the repository or branch that a particular OpenStack service
is created from, simply change the value of ``*_REPO`` or ``*_BRANCH``
corresponding to that service.

After making changes to the repository or branch, if ``RECLONE`` is not
set in ``localrc`` it may be necessary to remove the corresponding
directory from ``/opt/stack`` to force git to re-clone the repository.

For example, to pull Nova from a proposed release candidate in the
primary Nova repository:

::

    NOVA_BRANCH=rc-proposed

To pull Glance from an experimental fork:

::

    GLANCE_BRANCH=try-something-big
    GLANCE_REPO=https://github.com/mcuser/glance.git

Notes stuff you might need to know
----------------------------------

Reset the Bridge
~~~~~~~~~~~~~~~~

How to reset the bridge configuration:

::

    sudo brctl delif br100 eth0.926
    sudo ip link set dev br100 down
    sudo brctl delbr br100

Set MySQL Password
~~~~~~~~~~~~~~~~~~

If you forgot to set the root password you can do this:

::

    mysqladmin -u root -pnova password 'supersecret'