During a bench cycle the “Starting threads” dots means the number running threads:
Cycle #1 with 10 virtual users
------------------------------
* setUpCycle hook: ... done.
* Current time: 2011-01-26T23:23:06.234422
* Starting threads: ........
During the cycle logging the green dots means a successful test while the red ‘F’ are for test failure:
* Logging for 10s (until 2011-01-26T23:23:16.360602): ......F......
During the stagging down the dots are the number of stopped threads:
* Waiting end of threads: .........
Error : COOKIE ERROR: Cookie domain "<DOMAINE>" doesn’t start with "."
Comment the lines in file /usr/lib/python2.6/site-packages/webunit-1.3.8-py2.6.egg/webunit/cookie.py:
#if domain[0] != '.':
# raise Error, 'Cookie domain "%s" doesn\'t start with "."' % domain
Error : COOKIE ERROR: Cookie domain "."<DOMAINE>" doesn’t match request host "<DOMAINE>"
Comment the lines in the file /usr/lib/python2.6/site-packages/webunit-1.3.8-py2.6.egg/webunit/cookie.py:
#if not server.endswith(domain):
# raise Error, 'Cookie domain "%s" doesn\'t match '
# 'request host "%s"'%(domain, server)
High load works fine for IO Bound test, not on CPU bound test. The test script must be light:
On 32b OS install psyco, it gives a 50% boost (aptitude install python-psyco on Debia/Ubuntu OS).
On multi CPU server, GIL is getting infamous, to get all the power you need to use CPU affinity taskset -c 0 fl-run-bench is always faster than fl-run-bench. Using one bench runner process per CPU is a work around to use the full server power.
Use multiple machine to perform the load, see the next section.
Bench result file can be merged by the fl-build-report command, but how to run multiple bencher ?
There are many ways:
Use the new distribute mode (still in beta), it requires paramiko and virtualenv:
sudo aptitude install python-paramiko, python-virtualenv
It adds 2 new command line options:
For instance to use 2 workers you can do something like this:
$ fl-run-bench -c 1:2:3 -D 5 -f --simple-fetch test_Simple.py Simple.test_simple --distribute --distribute-workers=node1,node2 -u http://target/
========================================================================
Benching Simple.test_simple
========================================================================
Access 20 times the main url
------------------------------------------------------------------------
Configuration
=============
* Current time: 2011-02-13T23:15:15.174148
* Configuration file: /tmp/funkload-demo/simple/Simple.conf
* Distributed output: log-distributed
* Server: http://node0/
* Cycles: [1, 2, 3]
* Cycle duration: 5s
* Sleeptime between request: from 0.0s to 0.0s
* Sleeptime between test case: 0.0s
* Startup delay between thread: 0.01s
* Workers :octopussy,simplet
* Preparing sandboxes for 2 workers.....
* Starting 2 workers..
* [node1] returned
* [node2] returned
* Received bench log from [node1] into log-distributed/node1-simple-bench.xml
* Received bench log from [node2] into log-distributed/node2-simple-bench.xml
# Now building the report
$ fl-build-report --html log-distributed/node1-simple-bench.xml log-distributed/node2-simple-bench.xml
Merging results files: ..
nodes: node1, node2
cycles for a node: [1, 2, 3]
cycles for all nodes: [2, 4, 6]
Results merged in tmp file: /tmp/fl-mrg-o0MI8L.xml
Creating html report: ...done:
/tmp/funkload-demo/simple/test_simple-20110213T231543/index.html
Note that the version of FunkLoad installed on nodes is defined in the configuration file:
[distribute]
log_path = log-distributed
funkload_location=http://pypi.python.org/packages/source/f/funkload/funkload-1.16.1.tar.gz
Using BenchMaster http://pypi.python.org/pypi/benchmaster
Using Fabric http://tarekziade.wordpress.com/2010/12/09/funkload-fabric-quick-and-dirty-distributed-load-system/
Old school pssh/Makefile:
# clean all node workspaces
parallel-ssh -h hosts.txt rm -rf /tmp/ftests/
# distribute tests
parallel-scp -h hosts.txt -r ftests /tmp/ftests
# launch a bench
parallel-ssh -h hosts.txt -t -1 -o bench “(cd /tmp/ftests&& make bench URL=http://target/)”
# get the results
parallel-slurp -h hosts.txt -o out -L results-date -u ‘+%Y%m%d-%H%M%S’ -r /tmp/ftests/report .
# build the report with fl-build-report, it supports the results merging
Simple example with percent of users:
import random
...
def testMixin(self):
if random.randint(1, 100) < 30:
# 30% writer
return self.testWriter()
else:
# 70% reader
return self.testReader()
Example with fixed number of users:
def testMixin(self):
if self.thread_id < 2:
# 2 importer threads
return self.testImporter()
elif self.thread_id < 16:
# 15 back office with sleep time
return self.testBackOffice()
else:
# front office users
return self.testFrontOffice()
Note that when mixing tests the detail report for each page is meaningless because you are mixing pages from multiple tests.
The report is in reStructuredText, the index.rst can be edited in text mode, to rebuild the html version:
rst2html --stylesheet=funkload.css index.rst --traceback > index.html
Charts are build with gnuplot the gplot script file are present in the report directory to rebuild the pages charts for instance:
gnuplot pages.gplot
Since FunkLoad 1.15 you can also use an org-mode output to edit or extend the report before exporting it as a PDF.
Here is a sample Makefile
CREDCTL := fl-credential-ctl credential.conf
MONCTL := fl-monitor-ctl monitor.conf
LOG_HOME := ./log
ifdef URL
FLOPS = -u $(URL) $(EXT)
else
FLOPS = $(EXT)
endif
ifdef REPORT_HOME
REPORT = $(REPORT_HOME)
else
REPORT = report
endif
all: test
test: start test-app stop
bench: start bench-app stop
start:
-mkdir -p $(REPORT) $(LOG_HOME)
-$(MONCTL) restart
-$(CREDCTL) restart
stop:
-$(MONCTL) stop
-$(CREDCTL) stop
test-app:
fl-run-test -d --debug-level=3 --simple-fetch test_app.py App.test_app $(FLOPS)
bench-app:
-fl-run-bench --simple-fetch test_app.py App.test_app -c 1:5:10:15:20:30:40:50 -D 45 -m 0.1 -M .5 -s 1 $(FLOPS)
-fl-build-report $(LOG_HOME)/app-bench.xml --html -o $(REPORT)
clean:
-find . "(" -name "*~" -or -name ".#*" -or -name "*.pyc" ")" -print0 | xargs -0 rm -f
It can be used like this:
make test
make test URL=http://override-url/
# add extra parameters to the FunkLoad command
make test EXT="-V"
make bench
You can use the PageObject and fluent interface patterns as in the Nuxeo DM tests to write test like this:
class MySuite(NuxeoTestCase):
def testMyScenario(self):
(LoginPage(self)
.login('Administrator', 'Administrator')
.getRootWorkspaces()
.createWorkspace('My workspace', 'Test ws')
.rights().grant('ReadWrite', 'members')
.view()
.createFolder('My folder', 'Test folder')
.createFile('My file', 'Test file', 'foo.pdf')
.getRootWorkspaces().deleteItem("My workspace")
.logout())
Subscribe to the freshmeat project: http://freshmeat.net/projects/funkload