Github user Adrianlzt provided a python-twisted alternative version of pyclustercheck per discussion on issue 7.
Due to sporadic performance issues noted with the original implementation in SimpleHTTPserver, the benchmarks which I’ve included as part of the project on github use mutli-mechanize library,
- cache time 1 sec
- 2 x 100 thread pools
- 60s ramp up time
- 600s total duration
- testing simulated node fail (always returns 503, rechecks mysql node on cache expiry)
- AMD FX(tm)-8350 Eight-Core Processor
- Intel 330 SSD
- local loop back test (127.0.0.1)
The SimpleHTTPServer instance faired as follows:
Right away we can see around 500TPS throughput, however as can be seen in both response time graphs there are “outlying” transactions, something is causing the response time to spike dramatically SimpleHTTPServer, how does the twisted alternative compare? (note the benchmarks are from the current HEAD with features re-added to avoid regression, caching and ipv6 support)
Ouch! We appear to have taken a performance hit, at least in terms of TPS -19% rough estimates however compare the response time graphs to find a much more consistent plot, we had outliers hitting near 70s for SimpleHTTP server, we’re always under 1s within twisted.
Great! So why isn’t this merged into the master branch as the main project and therfor bundled with Percona XtraDB Cluster (PXC)? The issue here is the required version of python-twisted; ipv6 support was introduced in issue 8 by user Nibler999 and to avoid regression I re-added support for ipv6 in this commit for twisted
ipv6 support for python-twisted is not in the version distributed to main server OS’s such as
- EL6 python-twisted 8.x
- Ubuntu 10.04 LTS python-twisted 11.x
What’s the issue here? Attempting to bind / listen to an ipv6 interface yields the following error: twisted.internet.error.CannotListenError: Couldn't listen on :::8000: [Errno -9] Address family for hostname not supported.
Due to this regression (breaking of ipv6 support) the twisted version can not at this time be merged into master, the twisted version however as can be seen from above is much more consistent and if you have the “cycles” to implement it (e.g. install twisted from pypy via pip / easy_install to get >= 12.x) and test it’s a promising alternative.
To illustrate this further the benchmark was made more gruling:
- 5 x 100 thread pools
- 60s ramp up
- 600s total duration
First the twisted results, note the initial spike is due to a local python issue where it locked up creating a new thread in multi-mechanize:
Now the SimpleHTTPServer results:
Oh dear, as the load increases clearly we get some stability issues inside SimpleHTTP server…
Also worth noting is the timeouts
- twisted:
grep 'timed out' results.csv | wc -l== 0 - SimpleHTTPServer:
grep 'timed out' results.csv | wc -l== 470
… in the case of increased load the twisted model performs far more consistently under the same test conditions when compared against SimpleHTTPServer. I include the multi-mechanize scripts as part of the project on GitHub – as such you can recreate these tests yourself and gauge the performance to see if twisted or SimpleHTTP suits your needs.
Hi,
Also note the initial reason to rewrite the pyclustercheck is because only one petition could be attended simultaneously, and with a little of bad luck (losing some packets) pyclustercheck get stucked until restarted.
P.S.: Adrianlzt, no Arianlzt 🙂