From KVM

Contents 1 Autotest Server Installation (KVM perspective) 1.1 Notes 1.2 Server Installation Steps 1.2.1 Install required packages 1.2.2 Get Autotest 1.2.3 Important server configuration for the KVM test 1.2.4 Set up MySQL 1.2.5 Install extra packages 1.2.6 Update Apache config 1.2.7 Update Autotest config files 1.2.8 Run DB migrations to set up DB schemas and initial data 1.2.9 Adding some initial frontend objects 1.2.10 SELinux Issues 1.2.11 Restart Apache 1.2.12 Test the server frontend 1.2.13 Start the scheduler 1.2.14 Update kvm test config files 1.3 Host Installation Steps 1.3.1 Setup password-less ssh connection from the server to this host 1.3.2 Execute basic kvm job 1.3.3 Analyze kvm job execution results 1.4 Autotest Server Troubleshooting Q&A 1.4.1 Status is queing 1.4.2 Status is pending 1.4.3 How-to change mod

Autotest Server Installation (KVM perspective)

Notes

• Currently supported OS platforms (both server and remote hosts) are: Debian/Ubuntu, Red Hat/Fedora
• It is recommended to have at least 50GB of free disk space on each remote host prior to execution (to test all guests).
• Make sure you have python 2.4 (or higher) installed.
• Do this all as root (or at least run the tests as root)
• This server installation was tested on Fedora 12 and and is based on autotest server installation instructions

Server Installation Steps

Install required packages

yum install git make wget python-devel unzip
yum install httpd mod_python Django mysql-server MySQL-python gnuplot python-crypto python-paramiko java-1.6.0-openjdk-devel
yum install numpy python-matplotlib libpng-devel freetype-devel python-imaging

Get Autotest

Recently KVM autotest got merged with autotest upstream, therefore all the steps we used to perform with KVM autotest needs to be performed with autotest upstream.

AUTOTESTDIR=<your preferred directory> #tested using /usr/local/autotest
cd ${AUTOTESTDIR}
git clone git://github.com/ehabkost/autotest.git
git checkout 189eb9f96b1c88ca9cecd064c746baaca59c2766

As of when this document was written (01-13-2010), autotest is going through some changes and it's better to use an older revision, the one I recommend is 189eb9f96b1c88ca9cecd064c746baaca59c2766.

Note: If your AUTOTESTDIR is not /usr/local/autotest, you may have some issues with the apache configuration.

Important server configuration for the KVM test

The way the kvm control file is organized right now (as of 01-28-2010), it requires the user to change one value on global_config.ini file, that should be at the top of the autotest tree. Please change the following configuration value:

[PACKAGES]
serve_packages_from_autoserv: True

By default, the value is True, please change it to False. To make a long story short, changing this value will make autoserv to copy all tests to the server before trying to execute the control file, and this is necessary for the kvm control file to run.

Set up MySQL

Firstly, make sure that mysql daemon is up and starts on each boot:

/sbin/service mysqld restart
chkconfig mysqld on 

Next,provide mysql server with password by running the following command:

mysqladmin -u root password YOUR_PASSWORD

To get a mysql prompt, type mysql -u root -p. The following commands will set up mysql with a read-only user called nobody and a user with full permissions called autotest:

create database autotest_web;
grant all privileges on autotest_web.* TO 'autotest'@'localhost' identified by 'some_password';
grant SELECT on autotest_web.* TO 'nobody'@'%';
grant SELECT on autotest_web.* TO 'nobody'@'localhost';
create database tko;
grant all privileges on tko.* TO 'autotest'@'localhost' identified by 'some_password';
grant SELECT on tko.* TO 'nobody'@'%';
grant SELECT on tko.* TO 'nobody'@'localhost';

Install extra packages

Run the build script to install necessary external packages (You will need unzip to be able to use setuptools). Currently, the packages it installs are setuptools, mysql-python, Django, numpy, matplotlib, paramiko, simplejson and GWT:

/usr/local/autotest/utils/build_externals.py

Always re-run this after a svn update if you notice it has changed, new dependencies may have been added. This is safe to rerun as many times as you want. It will only fetch and build what it doesn't already have. NOTE: Set the HTTP_PROXY environment variable to http://proxy:3128/ before running the above if your site requires a proxy to fetch urls.

Update Apache config

Edit /etc/httpd/conf/httpd.conf as follows:

Insert the following line:

Include "${AUTOTESTDIR}/apache/conf/all-directives"

Uncomment the following line:

AddHandler cgi-script .cgi

Update Autotest config files

Edit the following files to match the passwords you set earlier. In addition, change the hostname and the notify_email_from fields. You might also use a dedicated smtp server, without relying on the one you have running on the machine.

${AUTOTESTDIR}/global_config.ini
${AUTOTESTDIR}/shadow_config.ini

Run DB migrations to set up DB schemas and initial data

During the time span of the project, the autotest database went through design changes. In order to make it able for people running older versions to upgrade their databases, we have the concept of migration. Migration is nothing but starting from the initial database design until the latest one used by this specific version of the application.

database/migrate.py --database=AUTOTEST_WEB sync
database/migrate.py --database=TKO sync

Adding some initial frontend objects

Compile the Autotest web application and TKO frontend

/usr/local/autotest/utils/compile_gwt_clients.py -a

You will need to re-compile after any changes/syncs of the frontend/client pages.

Run Django's syncdb

${AUTOTESTDIR}/frontend/manage.py syncdb

Make everything world-readable, for Apache's sake

chmod -R o+r ${AUTOTESTDIR}
find ${AUTOTESTDIR} -type d | xargs chmod o+x
chmod o+x ${AUTOTESTDIR}/tko/*.cgi
sudo chmod -R o+r /usr/lib/python2.5/site-packages/django

Make sure that current database users have admin privileges

mysql -u root -p -e 'update autotest_web.auth_user set is_superuser=1;'

SELinux Issues

You may encounter issues with SELinux not allowing a section of the web UI to work when running in Enforcing Mode. In order to fix this, you can run the following commands to allow execution of the cgi scripts on your server.

semanage fcontext-a -t bin_t '${AUTOTESTDIR}/tko(/.*cgi)?'
restorecon ${AUTOTESTDIR}

Restart Apache

/sbin/service httpd restart

Test the server frontend

• You should be able to access the web frontend at http://localhost/afe/.

Start the scheduler

You can execute the babysitter scripter through, let's say, nohup or screen. It is important to remember that by design, it's better to create an 'autotest' user that can run the scheduler and communicate with the machines through ssh.

${AUTOTESTDIR}/scheduler/monitor_db_babysitter

Update kvm test config files

cd ${AUTOTESTDIR}/client/tests/kvm
cp kvm_cdkeys.cfg.sample kvm_cdkeys.cfg
cp kvm_address_pools.cfg.sample kvm_address_pools.cfg
cp kvm_tests.cfg.sample kvm_tests.cfg

The sample configuration file construct a very basic Fedora 11 test set (@fc11_kickstart). It will use kickstart to install Fedora 11, then do some basic boot and shutdown tests.

Edit the file to suit your testing needs.

The server is now ready to use. Please check out the following sections to learn how to configure remote hosts and execute the kvm test suite.

Host Installation Steps

Setup password-less ssh connection from the server to this host

On the server, create rsa key in the following way:

ssh-keygen -t rsa

On the host (client) run:

scp server_name:/root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
/etc/init.d/sshd restart

Execute basic kvm job

• Click the "Admin interface" link at the upper-right corner of the frontend. If you get permissions issues with this, go back and execute the update autotest_web.auth_user SQL above to give the default user permissions.
• Click "Labels" and then "Add label".
• Type a name for the label, and check the "Platform" box, then click "Save".
• Click "Home" at the upper-left corner, and click "Hosts" and then "Add host".
• Type in a hostname, add your platform to the "chosen labels", and click "Save".
• Click "Home", and click "Tests" and then "Add test".
• Fill in the following information:

• • Name: kvm-autotest
  • Test class: kernel
  • Test type: Client
  • Path: client/tests/kvm/control
  • Make sure the Experimental check box is unchecked

• Click "Save".
• Now click "Go to Frontend" at the upper-right corner.

You should now be able to create a job on the "Create Job" tab, submit it, and see it in the "Job List" and "View Job" tabs. You can also see the status of your host on the "Hosts" tab.

Analyze kvm job execution results

It is recommended to work with the new results interface provided by autotest allowing SQL query based filtering, usable display of logs and test attributes and graphing capabilities. However, kvm-autotest also produces a detailed, formatted html report (results.html) per remote host tested in addition to standard autotest logs, where kvm-autotest results data is nicely organized. The html reports are stored in the job main results directory (accessible via raw results logs link).

Autotest Server Troubleshooting Q&A

Status is queing

You need to re-run scheduler. If you work via SSH, do it inside "screen" to make it reliable. If you have followed the advice of running the babysitter script under screen you shouldn't see this.

Status is pending

Usually it is a result of scheduler crash due to lack of disk space on Autotest server.

How-to change mod

# /usr/local/autotest/cli/atest-host mod -y *host_name*