Gitiles
Code Review Sign In
spfactory.storpool.com / devstack / 4e07fdcba76a58fc2b21ce6670144312ddb73869 / . / docs / source / guides / multinode-lab.html
blob: a286954e7de4fce8868415877995232fe601dbd6 [file] [log] [blame]
Dean Troyer54b97322014-06-20 17:53:55 -0500[diff] [blame]1<!DOCTYPE html>
2<html lang="en">
3 <head>
4 <meta charset="utf-8">
5 <title>Multi-Node Lab Server Guide - DevStack</title>
6 <meta name="description" content="">
7 <meta name="author" content="">
8
9 <!-- Le HTML5 shim, for IE6-8 support of HTML elements -->
10 <!--[if lt IE 9]>
11 <script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
12 <![endif]-->
13
14 <!-- Le styles -->
15 <link href="../assets/css/bootstrap.css" rel="stylesheet">
16 <link href="../assets/css/local.css" rel="stylesheet">
17 <style type="text/css">
18 body { padding-top: 60px; }
19 dd { padding: 10px; }
20 </style>
21
22 <!-- Le javascripts -->
23 <script src="../assets/js/jquery-1.7.1.min.js" type="text/javascript" charset="utf-8"></script>
24 <script src="../assets/js/bootstrap.js" type="text/javascript" charset="utf-8"></script>
25 </head>
26
27 <body>
28
29 <div class="navbar navbar-fixed-top">
30 <div class="navbar-inner">
31 <div class="container">
32 <a class="brand" href="/">DevStack</a>
33 <ul class="nav pull-right">
34 <li><a href="../overview.html">Overview</a></li>
35 <li><a href="../changes.html">Changes</a></li>
36 <li><a href="../faq.html">FAQ</a></li>
37 <li><a href="http://github.com/openstack-dev/devstack">GitHub</a></li>
38 <li><a href="https://review.openstack.org/#/q/status:open+project:openstack-dev/devstack,n,z">Gerrit</a></li>
39 </ul>
40 </div>
41 </div>
42 </div>
43
44 <div class="container">
45
46 <section id="overview">
47 <h1>Multi-Node Lab: Serious Stuff</h1>
48 <p>Here is OpenStack in a realistic test configuration with multiple physical servers.</p>
49 </section>
50
51 <section id="prerequisites">
52 <div class="page-header">
53 <h2>Prerequisites <small>Linux & Network</small></h2>
54 </div>
55
56 <h3>Minimal Install</h3>
Dean Troyer1c425992014-07-28 17:51:13 -0500[diff] [blame]57 <p>You need to have a system with a fresh install of Linux. You can download the <a href="https://help.ubuntu.com/community/Installation/MinimalCD">Minimal CD</a> for Ubuntu releases since DevStack will download &amp; install all the additional dependencies. The netinstall ISO is available for <a href="http://mirrors.kernel.org/fedora/releases/18/Fedora/x86_64/iso/Fedora-20-x86_64-netinst.iso">Fedora</a> and <a href="http://mirrors.kernel.org/centos/6.5/isos/x86_64/CentOS-6.5-x86_64-netinstall.iso">CentOS/RHEL</a>.</p>
Dean Troyer54b97322014-06-20 17:53:55 -0500[diff] [blame]58
59 <p>Install a couple of packages to bootstrap configuration:</p>
60 <pre>apt-get install -y git sudo || yum install -y git sudo</pre>
61
62 <h3>Network Configuration</h3>
63 <p>The first iteration of the lab uses OpenStack's FlatDHCP network controller so
64 only a single network will be required. It should be on its own subnet without DHCP;
65 the host IPs and floating IP pool(s) will come out of this block. This example
66 uses the following:</p>
67 <ul>
68 <li>Gateway: 192.168.42.1</li>
69 <li>Physical nodes: 192.168.42.11-192.168.42.99</li>
70 <li>Floating IPs: 192.168.42.128-192.168.42.254</li>
71 </ul>
72 <p>Configure each node with a static IP.
73 For Ubuntu edit <code>/etc/network/interfaces</code>:</p>
74
75 <pre>auto eth0
76iface eth0 inet static
77 address 192.168.42.11
78 netmask 255.255.255.0
79 gateway 192.168.42.1
80</pre>
81 <p>For Fedora and CentOS/RHEL edit
82 <code>/etc/sysconfig/network-scripts/ifcfg-eth0</code>:</p>
83
84 <pre>BOOTPROTO=static
85IPADDR=192.168.42.11
86NETMASK=255.255.255.0
87GATEWAY=192.168.42.1
88</pre>
89
90<!-- save these for the VLAN version
91 auto eth0.926
92 iface eth0.926 inet static
93 address 10.4.144.3
94 netmask 255.255.240.0
95 #gateway 10.4.144.1
96 up ifconfig eth0.926
97
98 auto eth0.605
99 iface eth0.605 inet static
100 address 172.16.103.3
101 netmask 255.255.255.0
102 #gateway 172.16.103.1
103 up ifconfig eth0.605 up
104-->
105
106 </section>
107
108 <section id="installation">
109 <div class="page-header">
110 <h2>Installation <small>shake and bake</small></h2>
111 </div>
112
113 <h3>Add the DevStack User</h3>
114 <p>OpenStack runs as a non-root user that has sudo access to root. There is nothing special
115 about the name, we'll use <code>stack</code> here. Every node must use the same name and
116 preferably uid. If you created a user during the OS install you can use it and give it
117 sudo priviledges below. Otherwise create the stack user:</p>
118 <pre>groupadd stack
119useradd -g stack -s /bin/bash -d /opt/stack -m stack</pre>
120 <p>This user will be making many changes to your system during installation and operation
121 so it needs to have sudo priviledges to root without a password:</p>
122 <pre>echo "stack ALL=(ALL) NOPASSWD: ALL" &gt;&gt; /etc/sudoers</pre>
123 <p>From here on use the <code>stack</code> user. <b>Logout</b> and <b>login</b> as the
124 <code>stack</code> user.</p>
125
126 <h3>Set Up Ssh</h3>
127 <p>Set up the stack user on each node with an ssh key for access:</p>
128 <pre>mkdir ~/.ssh; chmod 700 ~/.ssh
129echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyYjfgyPazTvGpd8OaAvtU2utL8W6gWC4JdRS1J95GhNNfQd657yO6s1AH5KYQWktcE6FO/xNUC2reEXSGC7ezy+sGO1kj9Limv5vrvNHvF1+wts0Cmyx61D2nQw35/Qz8BvpdJANL7VwP/cFI/p3yhvx2lsnjFE3hN8xRB2LtLUopUSVdBwACOVUmH2G+2BWMJDjVINd2DPqRIA4Zhy09KJ3O1Joabr0XpQL0yt/I9x8BVHdAx6l9U0tMg9dj5+tAjZvMAFfye3PJcYwwsfJoFxC8w/SLtqlFX7Ehw++8RtvomvuipLdmWCy+T9hIkl+gHYE4cS3OIqXH7f49jdJf jesse@spacey.local" > ~/.ssh/authorized_keys</pre>
130
131 <h3>Download DevStack</h3>
132 <p>Grab the latest version of DevStack from github:</p>
133 <pre>git clone https://github.com/openstack-dev/devstack.git
134cd devstack</pre>
135
136 <p>Up to this point all of the steps apply to each node in the cluster. From here on
137 there are some differences between the cluster controller (aka 'head node') and the
138 compute nodes.</p>
139
140 <h3>Configure Cluster Controller</h3>
141 <p>The cluster controller runs all OpenStack services. Configure the cluster controller's DevStack in <code>local.conf</code>:</p>
142 <pre>[[local|localrc]]
143HOST_IP=192.168.42.11
144FLAT_INTERFACE=eth0
145FIXED_RANGE=10.4.128.0/20
146FIXED_NETWORK_SIZE=4096
147FLOATING_RANGE=192.168.42.128/25
148MULTI_HOST=1
149LOGFILE=/opt/stack/logs/stack.sh.log
150ADMIN_PASSWORD=labstack
151MYSQL_PASSWORD=supersecret
152RABBIT_PASSWORD=supersecrete
153SERVICE_PASSWORD=supersecrete
154SERVICE_TOKEN=xyzpdqlazydog</pre>
155<!-- save for vlan
156FLAT_INTERFACE=eth0.926
157-->
158
159 <p>In the multi-node configuration the first 10 or so IPs in the private subnet are usually reserved. Add this to <code>local.sh</code> to have it run after every <code>stack.sh</code> run:</p>
160 <pre>for i in `seq 2 10`; do /opt/stack/nova/bin/nova-manage fixed reserve 10.4.128.$i; done</pre>
161
162 <p>Fire up OpenStack:</p>
163 <pre>./stack.sh</pre>
164 <p>A stream of activity ensues. When complete you will see a summary of
165 <code>stack.sh</code>'s work, including the relevant URLs, accounts and passwords to poke at your
166 shiny new OpenStack. The most recent log file is available in <code>stack.sh.log</code>.</p>
167
168 <h3>Configure Compute Nodes</h3>
169 <p>The compute nodes only run the OpenStack worker services. For additional machines, create a <code>local.conf</code> with:</p>
170 <pre>HOST_IP=192.168.42.12 # change this per compute node
171FLAT_INTERFACE=eth0
172FIXED_RANGE=10.4.128.0/20
173FIXED_NETWORK_SIZE=4096
174FLOATING_RANGE=192.168.42.128/25
175MULTI_HOST=1
176LOGFILE=/opt/stack/logs/stack.sh.log
177ADMIN_PASSWORD=labstack
178MYSQL_PASSWORD=supersecret
179RABBIT_PASSWORD=supersecrete
180SERVICE_PASSWORD=supersecrete
181SERVICE_TOKEN=xyzpdqlazydog
182DATABASE_TYPE=mysql
183SERVICE_HOST=192.168.42.11
184MYSQL_HOST=192.168.42.11
185RABBIT_HOST=192.168.42.11
186GLANCE_HOSTPORT=192.168.42.11:9292
Robbie Harwood (frozencemetery)4e07fdc2014-07-14 18:11:39 -0400[diff] [blame^]187ENABLED_SERVICES=n-cpu,n-net,n-api,c-sch,c-api,c-vol
188NOVA_VNC_ENABLED=True
189NOVNCPROXY_URL="http://192.168.42.11:6080/vnc_auto.html"
190VNCSERVER_LISTEN=$HOST_IP
191VNCSERVER_PROXYCLIENT_ADDRESS=$VNCSERVER_LISTEN
192</pre>
193
Dean Troyer54b97322014-06-20 17:53:55 -0500[diff] [blame]194<!-- save for vlan
195FLAT_INTERFACE=eth0.926
196-->
197
198 <p>Fire up OpenStack:</p>
199 <pre>./stack.sh</pre>
200 <p>A stream of activity ensues. When complete you will see a summary of
201 <code>stack.sh</code>'s work, including the relevant URLs, accounts and passwords to poke at your
202 shiny new OpenStack. The most recent log file is available in <code>stack.sh.log</code>.</p>
203
204 <h3>Cleaning Up After DevStack</h3>
205 <p>Shutting down OpenStack is now as simple as running the included <code>unstack.sh</code> script:</p>
206 <pre>./unstack.sh</pre>
207
208 <p>A more aggressive cleanup can be performed using <code>clean.sh</code>. It removes certain troublesome packages and attempts to leave the system in a state where changing the database or queue manager can be reliably performed.
209 <pre>./clean.sh</pre>
210
211 <p>Sometimes running instances are not cleaned up. DevStack attempts to do this when it
212 runs but there are times it needs to still be done by hand:</p>
213 <pre>sudo rm -rf /etc/libvirt/qemu/inst*
214sudo virsh list | grep inst | awk '{print $1}' | xargs -n1 virsh destroy</pre>
215
216 </section>
217
218 <section id="options">
219 <div class="page-header">
220 <h2>Options <small>pimp your stack</small></h2>
221 </div>
222
223 <h3>Additional Users</h3>
224 <p>DevStack creates two OpenStack users (<code>admin</code> and <code>demo</code>) and two tenants (also <code>admin</code> and <code>demo</code>). <code>admin</code> is exactly what it sounds like, a priveleged administrative account that is a member of both the <code>admin</code> and <code>demo</code> tenants. <code>demo</code> is a normal user account that is only a member of the <code>demo</code> tenant. Creating additional OpenStack users can be done through the dashboard, sometimes it is easier to do them in bulk from a script, especially since they get blown away every time
225 <code>stack.sh</code> runs. The following steps are ripe for scripting:</p>
226 <pre># Get admin creds
227. openrc admin admin
228
229# List existing tenants
230keystone tenant-list
231
232# List existing users
233keystone user-list
234
235# Add a user and tenant
236NAME=bob
237PASSWORD=BigSecrete
238TENANT=$NAME
239keystone tenant-create --name=$NAME
240keystone user-create --name=$NAME --pass=$PASSWORD
241keystone user-role-add --user-id=&lt;bob-user-id&gt; --tenant-id=&lt;bob-tenant-id&gt; --role-id=&lt;member-role-id&gt;
242# member-role-id comes from the existing member role created by stack.sh
243# keystone role-list</pre>
244
245 <h3>Swift</h3>
246 <p>Swift requires a significant amount of resources and is disabled by default in DevStack.
247 The support in DevStack is geared toward a minimal installation but can be used for
248 testing. To implement a true multi-node test of Swift required more than DevStack provides.
249 Enabling it is as simple as enabling the <code>swift</code> service in <code>local.conf</code>:
250 <pre>enable_service swift</pre>
251
252 <p>Swift will put its data files in <code>SWIFT_DATA_DIR</code> (default <code>/opt/stack/data/swift</code>).
253 The size of the data 'partition' created (really a loop-mounted file) is set by
254 <code>SWIFT_LOOPBACK_DISK_SIZE</code>. The Swift config files are located in
255 <code>SWIFT_CONFIG_DIR</code> (default <code>/etc/swift</code>). All of these settings can be overridden in
256 (wait for it...) <code>local.conf</code>.</p>
257
258 <h3>Volumes</h3>
259 <p>DevStack will automatically use an existing LVM volume group named <code>stack-volumes</code>
260 to store cloud-created volumes. If <code>stack-volumes</code> doesn't exist, DevStack
261 will set up a 5Gb loop-mounted file to contain it. This obviously limits the
262 number and size of volumes that can be created inside OpenStack. The size can be
263 overridden by setting <code>VOLUME_BACKING_FILE_SIZE</code> in <code>local.conf</code>.</p>
264
265 <p><code>stack-volumes</code> can be pre-created on any physical volume supported by
266 Linux's LVM. The name of the volume group can be changed by setting <code>VOLUME_GROUP</code>
267 in <code>localrc</code>. <code>stack.sh</code> deletes
268 all logical volumes in <code>VOLUME_GROUP</code> that begin with
269 <code>VOLUME_NAME_PREFIX</code> as part of cleaning up from previous runs.
270 It is recommended to not use the root volume group as <code>VOLUME_GROUP</code>.</p>
271
272 <p>The details of creating the volume group depends on the server hardware involved
273 but looks something like this:</p>
274 <pre>pvcreate /dev/sdc
275vgcreate stack-volumes /dev/sdc</pre>
276
277 <h3>Syslog</h3>
278 <p>DevStack is capable of using <code>rsyslog</code> to agregate logging across the cluster.
279 It is off by default; to turn it on set <code>SYSLOG=True</code> in <code>local.conf</code>.
280 <code>SYSLOG_HOST</code> defaults to <code>HOST_IP</code>; on the compute nodes it
281 must be set to the IP of the cluster controller to send syslog output there. In the example
282 above, add this to the compute node <code>local.conf</code>:</p>
283 <pre>SYSLOG_HOST=192.168.42.11</pre>
284
285 <h3>Using Alternate Repositories/Branches</h3>
286 <p>The git repositories for all of the OpenStack services are defined in <code>stackrc</code>.
287 Since this file is a part of the DevStack package changes to it will probably be overwritten
288 as updates are applied. Every setting in <code>stackrc</code> can be redefined in
289 <code>local.conf</code>.</p>
290
291 <p>To change the repository or branch that a particular OpenStack service is created from,
292 simply change the value of <code>*_REPO</code> or <code>*_BRANCH</code> corresponding to
293 that service.</p>
294
295 <p>After making changes to the repository or branch, if <code>RECLONE</code> is not set
296 in <code>localrc</code> it may be necessary to remove the corresponding directory from
297 <code>/opt/stack</code> to force git to re-clone the repository.</p>
298
299 <p>For example, to pull Nova from a proposed release candidate in the primary Nova
300 repository:</p>
301 <pre>NOVA_BRANCH=rc-proposed</pre>
302
303 <p>To pull Glance from an experimental fork:</p>
304 <pre>GLANCE_BRANCH=try-something-big
305GLANCE_REPO=https://github.com/mcuser/glance.git</pre>
306
307 </section>
308
309 <section id="Notes">
310 <div class="page-header">
311 <h2>Notes <small>stuff you might need to know</small></h2>
312 </div>
313
314 <h3>Reset the Bridge</h3>
315 <p>How to reset the bridge configuration:</p>
316 <pre>sudo brctl delif br100 eth0.926
317sudo ip link set dev br100 down
318sudo brctl delbr br100</pre>
319<!--
320 sudo rm /etc/libvirt/qemu/*.xml
321-->
322
323 <h3>Set MySQL Password</h3>
324 <p>If you forgot to set the root password you can do this:</p>
325 <pre>mysqladmin -u root -pnova password 'supersecret'</pre>
326
327 </section>
328
329 <footer>
330 <p>&copy; Openstack Foundation 2011-2014 &mdash; An <a href="https://www.openstack.org/">OpenStack</a> <a href="https://wiki.openstack.org/wiki/Programs">program</a></p>
331 </footer>
332
333 </div> <!-- /container -->
334
335 </body>
336</html>