Redis has simple single process event driven design, which means it does not have to deal with any locks which is performance killer for a lot of applications. This however limits it scalability to single core. Still with 100K+ operations a second this single core performance will be good enough for many applications. Also nothing stops you from running many Redis instance on single server to get advantage of multiple cores.

I call Redis semi-persistent because it does not store the data on disk immediately but rather dumps its all database every so often – you have a choice of configuring time and number of updates between database dumps. Because dump is basically serial write Redis does not an expensive IO subsystem. Also because this dump is background it does not affect read/write performance to the database which is in memory. In the tests I’ve done I’ve seen Redis doing writes some 4MB/sec for probably 50% of test duration where Innodb had to write 50MB/sec for about third of throughput and doing a lot of random IO as it was doing it. This is among other things because Innodb has to flush full 16K pages while doing flush.

The background flush in Redis is designed the following way – Redis process forks and justs dumps the database it has in the background. Unix copy on write takes care of getting another copy of pages as they are modified. This keeps overhead rather low. The database is dumped in temporary file which is renamed only after fsync which means if you crash during the dump you simply discard partial file.

I also liked full pipelining support in the protocol – you can send multiple commands at once – any commands and redis server will process them in order and returns results to you. This not only allows for multi-get and multi-set but any of batches of commands being submitted. The API support for this is relatively week but the features are there.

When it comes to data types – Redis supports simple key-value storage just as memcache but it also adds support for Lists, which are similar to the linked list data type you would have as well as sets which allow to store sets of strings with support of various set operations.

I kind of miss support for something like associative array/hash/map in the data types but I guess there is nothing in architecture which would stop it from being added later.

Redis also has support for “databases” which are basically key spaces inside the server. This should allow for using server by different applications, different versions, testing or some other features. Though you’ve got to be careful with these – there is only simple per instance password based authentication in place, so if application can talk to the instance it can access all databases.

I am also a bit surprised why databases are numbered instead of named. Naming the databases would make it more simple to avoid unwanted conflicts etc.

Redis also supports master/slave replication out of the box and it is extremely simple. You just specify from which node to replicate and this is it. It is even more simple than with MySQL as you do not need to deal with snapshot or binary log position. Replication is asynchronous and low overhead – redis will perform the database dump and store the commands on the data since the start of the process. Slave can get the data (which is basically set of commands to populate database itself) and when get the data from the master as it comes. I did not benchmark the replication capacity but I’d expect it to be close to 100K of writes/sec the single instance can handle.

The benchmarks I’ve done were for applications which is very update intensive with updates being pretty much random single row updates which are hard to batch. With MySQL/Innodb I got server being able to handle some 30.000 updates/sec on 16 core server with replication being able to handle 10.000 updates/sec. This was using about 5 cores so you could probably get 4 MySQL instances on this server and get up to 100K updates/sec with up to 40K updates/sec being able to replicate.

With Redis I got about 3 times more updates/sec – close to 100.000 updates/sec with about 1.5 core being used. I have not tried running multiple instances and I’m not sure the network and TCP stack would scale linearly in this case but anyway we’re speaking about hundreds of thousands of updates/sec.

I think Redis can be great piece of architecture for number of applications. You can use it as the database or as cache (it supports data expiration too)

I have not benchmarked it against memcache in terms of performance and memory usage. This may be another project to look at.

Entry posted by peter | 21 comments

Add to: | | | |

Implement proper error control In reality it took some time to find what was the issue because there was no error reporting for situation of unavailable web services. If log would be flooded with messages about web services being unavailable it would be much easier to find.

User Curl PHP Has a lot of functions which can accept URL as parameter and just fetch the data transparently for you. They however do not give you good error control and timeout management compared to curl module. Use that when possible it is easy. You can implement your own class to fetch required URL with single call while having all needed timeout handling and reporting to match your application needs. If you’re using PHP functions make sure default_socket_timeout has proper value or set it per session.

Set Curl Timeouts Set both TIMEOUT and CONNECT_TIMEOUT as these apply to different connection stages and just setting timeout is not enough.

Beware of PHP sessions “files” handler I already wrote about this topic, but when troubleshooting this all takes another angle. Default file handler means file gets locked while PHP request is being served. In this case because of network stall request could be taking 100+ seconds. Users are inpatient and do not wait so long pressing reload multiple times… which just adds to the list of users waiting on session file lock. This not only makes apache slots consumed at much higher pace but makes it harder to find what exactly is causing the lock because most of offending processes you can find from apache “server-status” will be just waiting on file to be unlocked. I used “gdb” to connect to the process showing high number of seconds since start finding where it is stuck. If it is somewhere in curl module (or mysql – waiting on long query to come back) – this is our query if it is waiting on the session file lock we can get that file and use fuser to see what other processes are using that files – these would be either waiting on locks or owning the lock and so one of them is the process we’re looking for. Things are much easier with say memcached session storage – this does not cause any locks for parallel session use so only the process which actually stalls waiting on external resource will show high number of seconds since request start.

Entry posted by peter | 5 comments

Add to: | | | |

I had already written about the mess that 64-bit integers are in PHP. But if the numbers you use do not cover 64-bit range fully, floats might save the day. The trick is that PHP floats are in fact doubles, i.e. double-precision 64-bit numbers. They have 52 bits for mantissa, and integer values up to 2^53-1 can be stored exactly. So if you're using up to 53 bits, you're OK with floats.

However, there's a conversion caveat you should be aware of.

On different systems, float is converted to string differently. (I spent a bit of time fighting with it today.) Consider the following script:

It prints the following (normally expected) output on x64 box running Linux with PHP 5.1.6 or 5.2.2, and also with PHP 4.4.2 on SPARC64 under NetBSD 3.0:

However 32-bit FreeBSD with PHP 4.4.7 produces different results:

The same 32-bit FreeBSD box with PHP 5.2.5 starts to emit all digits but still incorrectly:

And PHP 5.2.2 on Windows produces yet another variant:

As you can see float to string conversion is not portable across systems and PHP versions. So if you're handling large numbers stored as float (especially on 32-bit systems where 32-bit int overflow will implicitly convert to float) and getting strange bugs, remember that string conversion might let you down. sprintf(), on the other hand, is always a friend when it comes to fixing PHP "subtleties" in numeric value handling: it can be used to workaround signed vs. unsigned int issues; it helps with float formatting; always a saviour.

UPDATE: as Jakub Vrana pointed out in the comments (thanks!), it's the "precision" option set from php.ini which affects the conversion. I've played with it a bit; the compiled-in and php.ini-dist values seem to vary across architectures and versions, but setting precision=16 (enough to hold 2^53 in decimal without sign) helped in all cases.

However at the moment the option is neither mentioned in the online documentation, nor explained clearly in php.ini. So this adds another fixing route (ie. you could just set higher precision); but the caveat is still there.

Entry posted by shodan | 18 comments

Add to: | | | |

First I am going to describe the setup a little bit. As I already mentioned it was about web servers. Each of them was running Lighttpd that handled the requests coming from the internet. It was configured however only to serve static content, such as images. The requests asking for PHP files were passed down with proxy module to Apache listening on another TCP port.

And so I started investigating the problem. As it turned out the systems were slowing down because lighttpd process grew to a few gigabytes eating the entire memory which caused system to start swapping heavily. This usually means death to a busy on-line system. Initially I thought about hitting some Lighttpd bug as nothing else seemed wrong, but after a short while I remembered one thing that can cause such behavior. If you use it as a proxy, it will need to buffer the entire response from the backend server before sending to the client. And indeed I started browsing Apache access log and found entries similar to this appearing every few minutes:

Apparently some bug in PHP code with a loop having far too many iterations and even though Apache and PHP handled it without much hassle, Lighttpd kept allocating memory to fit the entire response into the buffer and caused all that mess.

Although this was an extreme case, it is easy to imagine the situation where the problems will appear with much smaller data being sent through Lighttpd proxy. For example with a PHP script for handling larger file downloads which gets many concurrent requests.

Entry posted by Maciej Dobrzanski | 10 comments

Add to: | | | |

The count of forum posts being large, is however not the largest we've got to deal in the project - number of links originating from forum posts is a bit larger number.

The task we had for links is to be able to search for links pointing to given web site as well. The challenge in this case is we do not only want to match links directed to "mysql.com" but links to "www.mysql.com" or "dev.mysql.com/download/" as well as they are all considered to belong to mysql.com domain, while searching for "dev.mysql.com/download"" will only match dev.mysql.com domain and files within /download/ directory.

Initially we implemented it in MySQL using partitioning by domain which link was pointing to. So "mysql.com" links were stored in one table group and "google.co.uk" on another. We still had serious challenges however - as each applies to many search URLS,
such as "dev.mysql.com/download/mysql-5.1.html" would match "mysql.com", "dev.mysql.com", "dev.mysql.com/download/" and
"dev.mysql.com/download/mysql-5.1.html" we could not use link=const where clause but had to use link like "prefix%" which means index could not be used to get 20 last links and filesort over millions of links we had to youtube.com wikipedia.org and other top domains was extremely slow. Not to mention counting number of links (and number number of distinct forum sites) pointing to the given URL or graphs showing number of links per day. To fight this problem we had to restrict number of days we allow to cover based on the amount of links to the domain... but for some top domains it was slow even with just 3 days worth of data.

You might point out if we had link_date between X and Y and link like "prefix%" kind of where clause we would not be able to use index past link_date part, it is true so we had to use link_date in ( ) and link like "prefix%" which allows to use both keyparts which is much better but not good enough.

Caching is not good enough in such case as we do not want a single user to wait for minutes. large variety of problematic search urls does not allow to use pre-caching not to mention general load on server such batch processing would put.

The first alternative to this approach was to store duplicate data storing link to "dev.mysql.com/download/mysql-5.1.html" as links to 4 url prefixes I mentioned above. Unfortunately this would blow up data stored quite significantly, requiring in average of 6 rows for each link and it does not solve all the problems - result counting and number of distinct sites were still pretty slow and we did not want to go into creating all this data as summary tables.

Instead we decided to use Sphinx for this kind of task which proved to be extremely good idea. We converted all URLs to search keywords and now these 6 rows become simply one row in sphinx index with 6 "keywords" - specially crafter strings which corresponded to the URLs. Of course we did not store these in the table but instead used UDF to convert URL to list of "keywords" on the fly.

As results we now can pull up results even for youtube.com for fractions of the second and we could show 3 months worth of data for any URLs. (We could use longer time span but we did not have enough memory for Sphinx attribute storage). It is especially great as there is still room for optimization - Sphinx stores word positions in the index, while we do not need them in this case as we're doing kind of "boolean full text search". Plus we can make index built sorted by timestamp which would allow to same on sorting which is now still happening.

Using Sphinx such non-traditional way required implementing some features more traditional for SQL databases rather than full text search applications. Group By was added to Sphinx so we could search number of matches per day, or number of matches per language.

For Domain Profile we've got to use even more of those features such as counting number of distinct sites pointing to the given url or domain etc. Honestly this is where we cheated a bit and distinct number is bit approximate for large numbers but it still works really well for our needs.

Sure we could use summary tables for domains but it would be a lot of work and raver inflexible if we would like to add some more features and take a lot of resources to update or periodically build for millions of domains.

As this part worked pretty well we also decided to use Sphinx for other part of the web site - Forum Site Profile. This uses some pre-generated data such as number of posts in total for forum or in thread but most of other stuff is built with Sphinx. This also uses fair amount of tricks using fake full text search to retrieve all posts from given web site or forum from the global sphinx index.

So in general we find parallel processing using sphinx pretty good solution for many data crunching needs especially when lack of parallel abilities in MySQL makes its use rather inconvenient and pre-generating data is inconvenient or impossible.

If you're attending OSCON 2007 and would like to learn more about Sphinx we have a BOF on Thursday to talk on the topic.

Entry posted by peter | 15 comments

Add to: | | | |

One of the process web site had was creating of summary tables which was done by executing some huge group by query, doing some stuff with results and then populating tables. This all worked well for small tables... but not for larger ones.

First problem was PHP script generating the table took 10GB of RAM and was swapping development server which had just 4GB of Ram (and plenty of swap space) like crazy. Why ? Because by default mysql_query uses mysql_store_result C library call and buffers all result set in the process memory. Not good if there are over 50 millions of rows. Note this limit is not controlled by memory_limit PHP config variable because that only controls memory which passes via PHP memory management which does not apply to MySQL result set.

OK there is "easy" fix for this problem, you can use mysql_unbuffered_query instead and mysqli and PDO have their own way to reach similar behavior. This call users underlying mysql_use_result API call which does not store all result set in memory but instead streams it from the server, fetching in blocks. There are some limits as you can't use mysql_num_rows() and mysql_data_seek() if you use this method but this is told in PHP manual and so easy to catch. There are however more differences which may cause things breakage

1. Table Locks - Table locks are not cleared until you fetch whole result set if you're reading from tables directly (if you do not have "using temporary" in EXPLAIN) this was not issue for given case as GROUP BY in question required temporary table plus it was test system anyway. The workaround for this one is to use SQL_BUFFER_RESULT hint if you need to release table locks early. It comes at cost of creating temporary table though which can be quite high.
2. Sharing connection no more works If you use buffered query you can use same connection to run other queries, ie INSERTs and UPDATEs while you traverse with data. Not with unbuffered query because connection is still busy.
3. Need more error checking If you use buffered query the only real call you can get errors is when you run mysql_query , mysql_fetch_row simply reads data from memory and so most applications do not care to check if there are any errors while fetching. With mysql_unbuffered_query data comes in portions so you can well get an error while fetching rows. If you do not check for error it can look as you've done with result set while you only processed a portion of it, which can cause rather hard to catch errors.
4. Connection can timeout If you do not fetch data for long enough MySQL Server may think client is dead and close connection. This well may happen if you need long processing for each row or have long periodic data flushes, ie with multiple value INSERTs etc. This can be fixed by increasing net_write_timeout variable on the server so it gives you more time

But is this the only way ?

Of course not. First you should consider if you need to do processing in PHP at all. Many summary tables can be built by INSERT ... SELECT, or some others purely SQL commands and it can be much more efficient. Another alternative is of course to use MySQL Stored Procedures which can be fit to do this simple job.

The downside of using these techniques if of course you've got to have summary tables and original tables on the same server which can limit your scalability. Using FEDERATED Tables can work for some cases in others script can be more efficient especially when multiple servers are involved and you want to do some parallel processing.

It is also good question if you need to query all result at once. It is rather efficient bur can cause problems with table locks and other issues plus if script aborts it may be hard to restart. So it may be better structuring your queries to process data by certain objects (ie City by one City at the time) or do INSERT ... SELECT to the temporary table with auto_increment column and fetch data from this table using auto increment column ranges instead. I would especially recommend this last way for very long processes, ie if you need to check data against web services and so on - in this case the overhead of creating yet another temporary table is not so large.

Entry posted by peter | 6 comments

Add to: | | | |

After all, their development started a few decades after C, so its notorious caveats are all well-known and should be easy to avoid when designing a new language, right?

However, PHP just brought me a new definition of "portable" - and that was when working with... integers.

PHP is not able to handle unsigned integers, and converts values over 2^31 to signed. So if your IDs go slightly over 2 billion, and PHP decides to treat them as integers, you're in trouble.

Oh wait, no - that's on 32-bit platforms only! PHP int size is platform-dependent, and it seems to be 8 bytes on our 64-bit boxes. Yes, the very same ones where C/C++ int is 4 bytes, you know.

That was the easy part. It was mostly documented.

Now, there's a function called unpack() which essentially allows to convert different types of data from binary strings to PHP variables. What if you try to unpack unsigned 32-bit big endian integer (format code "N")? Let's check the doc:

If you specify a number beyond the bounds of the integer type, it will be interpreted as a float instead.

Having read the doc I personally blatantly relied upon it and expected that large unsigned 32bit numbers would be converted to float, or string, or something, but handled properly. However, a couple or so weeks ago the following notice suddenly appeared:

"Note that PHP internally stores integral values as signed. If you unpack a large unsigned long and it is of the same size as PHP internally stored values the result will be a negative number even though unsigned unpacking was specified."

How sweet. No, it just could not behave like documented and convert 32-bit unsigned value to float on x32 or keep it integer on x64 - you now suddenly have to care about value size yourself. Ah, and by the way, there's no official way to know what's int size.

To make things even better, 5.2.1 introduced a nice bug in unpack(), which f..ed unpacking less-than-16-bit values on x64. (I assume you understand that "f..ed" means "fixed"). It took some time and several tries to convince PHP team that x64 has enough bits to hold 16-bit unpacked value, but thankfully its now acknowledged and assigned.

To summarize, if you need to unpack an unsigned 32bit int from binary stream, you have to:

• convert it to float or string manually,
• do that depending on int size on current platform,
• which can not be done using anything documented,
• and specifically avoid PHP 5.2.1 on x64.

Most people could probably learn all that, and then use sprintf("%u",$id), work with string IDs everywhere, avoid 5.2.1 and be happy.

Unfortunately, my final goal was to have support for 64-bit document IDs...

Let's do a small time travel. Integer types in C/C++ have always been a pain, but back in 1999 ISO commitee ratified ISO/IEC 9899:1999 standard, also known as ISO C99, which guarantees that "long long int" integer type must be at least 64 bits in size. By now, most compilers support that part perfectly.

However, designers of PHP 5 (released in 2004) type system were either not aware of this change, or decided to not rely on the standard which has been out for "only" 5 years by then, or just thought that 31 (no typo) bits and 640K should be enough for everybody.

Long story short, it's 2007 now but there's no native 64-bit integer type in PHP. Let me remind that built-in "int" might be 64-bit, but then again it might be not, and there's no official way to tell.

This time, there's a number of routes one could take - either use ints (and pray that the app is never run on x32, and that "platform dependent" size does not change to 4 next version); or use GMP or bcmath extensions if they are available.

Fine, so 99.999% of the world would hit that, compile in bcmath, and be happy again.

Unfortunately, I needed to develop a library which could be deployed in any environment - and still work, and produce reasonable results. The worst case is x32, and neither GMP nor bcmath available.

And this is how the following code was born.

For reference, this is what would the equivalent C/C++ snippet look like:

Portability in year 2007.

Entry posted by shodan | 17 comments

Add to: | | | |

If you use file based sessions PHP will lock session file for whole script execution duration, which means all requests from the same sessions will be serialized on PHP level, which means they also will be serialized for single user on database level. If you change to store PHP sessions in MySQL instead this effect may be no more true and you may have number of requests executing for the same session at the same time. First of course means you may have your session data damaged because you will have lost session variables update from one of the script, in addition however you may run into database related issues of modifying user profile or other user/session related data in parallel, if you do not use transactions or lock tables.

So how you can get back your old file based session behavior with MySQL Sessions ?

If you have dedicated connection to session database and use Innodb tables for your session storage you can start transaction on the session start and use SELECT ... FOR UPDATE to lock the session row in the session table for whole request length. On the end of the session the same row is updated and transaction is committed.

If you share session connection with other modules or do not use transactional tables for session you can use GET_LOCK to get same behavior. In the start of the session you can do SELECT GET_LOCK('',10) and in in the end of the request
SELECT RELEASE_LOCK('') where session_id is current session identifier. Note - setting this external lock on session name should be done before session data is read from database for things to work properly.

This approach assumes you do not use GET_LOCK in other places in your application as as soon as it is called second time previous lock is automatically released. The good thing about it however - you can use it as an extra to your current MySQL Sessions system without need to change how it works internally. If you do not use persistent connections you even do not have to release lock - as soon as connection is closed the lock is automatically released.

The value 10 in GET_LOCK is timeout in seconds - if lock can't be granted for this amount of time it will return "0" indicating lock was not granted in this case you can select to continue without session or may do something else, like logging error as this generally should not happen in well tuned applications.

Entry posted by peter | 26 comments

Add to: | | | |

Note this list applies to static content distribution, dynamic content has some of its own issues which need different treatment.

DNS TTL Settings The system was using DNS based load balancing, using something like img23.domain.com to serve some of the images. I'm not big fan of purely DNS based load balancing and HA but it works if configured well. In this case however the problem was zero TTL set in DNS configuration. This obviously adds latency especially for "aggregate" pages which may require images to be pulled from 10 different image servers.

Keep Alive In my previous post I wrote you often do not need keep alive for dynamic pages (there are also exceptions) but you really should have Keep Alive enabled while serving images. It especially hurts not to have one if 30 thumbnails are loaded per page if you do not have one.

Use Proper Web Server This one is pretty interesting. Many learned apache is not good for serving static content especially with many thousands of keep-alive connections. Lighttpd is often named as faster alternative. It is surely true if you server static content from memory or serving large files (in which case read-ahead helps) but if you're serving many millions of thumbnails it may not work well. Lighttpd 1.4 is single threaded and that single threads is used to handle both network and IO which is not going to scale especially if you have storage build from large number of hard drives. This problem is fixed in Lighttpd 1.5 which is however still in pre-release (though we have it running in production). There are other solutions such as nginx which does not seems to have this problem. My point with this item is not advice you which exactly web server to use but to point out serving dynamic content, serving in-memory static content and serving static content from the disk are different tasks and should not be mixed.

Use noatime mount option Serving images you rarely need last access time tracked for them, especially as this tracking can be quite expensive if there are a lot of different files accessed concurrently. It is especially worth to note updating access time needs io even in case content is in OS cache. Simple solution is use noatime mount option for your static content partition. Some web servers also support for O_NOATIME file open flag on newer Linux versions, which also can be used.

Using PHP to serve files This usually comes in play if security control for files or traffic limit per user needs to be enforced. Serving file by simply reading from PHP or any other heavy programming language and sending it back is worse thing you can do. The optimal solution would be to using some server modules for access control or hacking one if you need some special functionality, few people however have skills to do this kind of job. The other solution is to use X-send-file or similar custom headers which make PHP script just to check restrictions and have web server to send the file. This technique actually was used in the project we're speaking about with exception of resuming file downloads. In lighttpd 1.4 you could not tell server to send only part of the file so it had to be implemented in php. However even small portion of such downloads caused a lot of trouble especially as lighttpd tries to buffer all content it gets from PHP script which means many megabytes for large files. Partial file sending happily was added in lighttpd 1.5 which was one of the reason to move to it quickly.

Thumbnail size In many cases you need multiple sizes of thumbnails, such as very small size to show in list mode, standard size to show in preview mode and may be something else. It looks very attractive to keep only one thumbnail and simply set browser image sizes if you need to stick it to smaller space. It is not so good in practice however. First browsers may not resize image with optimal quality and it may just look ugly compared to properly created thumbnails. It also makes things a lot slower. I for example seen 8K thumbnail in the case when smallest one should have been less than 1KB in size. With 30 images on the page it is 30KB worth of download vs 240KB which makes quite a bit of difference especially on slower connection speeds.

Forgetting to set expire Static content rarely changes, in many implementations it may never change as loading new version of the same object will result in different url. This means you really need to have expire headers for them set, otherwise you will be getting a lot of cache revalidation requests which still require stat() calls on web server side and which can be avoided. How far in the future you want expire to be... it depends on the application of course. It can range for few hours for objects changes to which you want to be quickly visible to infinity for objects which only change with their urls.

Server side caching The benefit of server side caching is not so obvious for serving static content, or better say it depends on situation. If you serve a lot of small files which can fit in squid (for example) in-memory cache it may work quite well and in fact will give better memory utilization as OS cache has single page granularity (meaning 100 bytes file will still take 4KB in cache). For serving large set of files (which does not fit in memory) you can have performance to go down as request will be frequently made to the main server to get the file. Also it frequently does not make sense to use disk cache for static content as getting it from the server may be close in speed. It also of course depends on the server which you're using - apache in prefork mode (ie same server used for static and dynamic content) would likely to benefit a lot from one.

Using different servers As I already mentioned serving different content requires different skills from web servers or different configuration, so it is good idea to use dedicated server name, such as static.domain.com for serving static content, so even if you do not need it now you have flexibility in the future. Also even for same server it allows to configure different virtual host with different settings easily.

Entry posted by peter | 7 comments

Add to: | | | |

I see if frequently with web applications and it is often indication of trouble. Not only it means you may run out of MySQL connections quicker than you expected but it also frequently indicates serious problems in the application. If you do not use persistent connections and you have connection in Sleep stage for 600 seconds what could it be ? It may mean some of your pages take that long to generate (or might be the code simply gets into the tight loop and page never gets generated) it also could mean some of external Web Services are slow or not available and you're not dealing with timeouts properly. Or may be you have several connections to MySQL server and right now running query which takes that long ? In any case it is something frequently worth looking at.

First task is to find to which process the connection belongs. Using different user names for different application is a good practice however it will not tell you which of apache children is handling request in question. If you just want to fix it, ie by restarting apache it is enough but if you want to figure our why it is happening you need more info.

You my notice in the "Host" filed of SHOW PROCESSLIST output not only host but also port is specified, showing you something like
"192.168.1.70:58555" This port can be used to identify the process which owns connection in question:

[code]
[root@w1 ~]# netstat -ntp | grep :45384
tcp 0 0 192.168.1.70:45384 192.168.1.82:3306 ESTABLISHED 28540/php-cgi
[\code]

As you can see in this case we can find php-cgi is holding connection in question (this is lighttpd based system with fastcgi)

Now you know the process and you can use your favorite tools to check what that process is doing.

[code]
[root@w1 ~]# netstat -ntp | grep 28540
tcp 0 0 192.168.1.70:58555 192.168.1.90:11211 ESTABLISHED 28540/php-cgi
tcp 0 0 192.168.1.70:52711 192.168.1.88:8080 ESTABLISHED 28540/php-cgi
tcp 0 0 192.168.1.70:45384 192.168.1.82:3306 ESTABLISHED 28540/php-cgi
tcp 0 0 192.168.1.70:45399 192.168.1.82:3306 ESTABLISHED 28540/php-cgi
tcp 0 0 192.168.1.70:45407 192.168.1.82:3306 ESTABLISHED 28540/php-cgi
tcp 0 0 192.168.1.70:45408 192.168.1.82:3306 ESTABLISHED 28540/php-cgi
tcp 0 0 192.168.1.70:35556 192.168.1.92:11211 ESTABLISHED 28540/php-cgi
[\code]

Using same netstat command and filtering on the PID we can find which connections does this process have. Here you can see it has couple of memcached connections. Few MySQL connections (to the same host, which if usually bad idea) and connection to some external web server.

You can use strace -p to see what host is doing, it often gives a clue. In this case I for example found the process is stuck in pool() system call reading from network. Using netstat can give you an idea what it can be but if you do not like guessing you can use gdb -p . It will not print you exact line of code in PHP which is running but can give you some good ideas - for example in this case I could find stack trace originated from php stream functions not from libmysql or memcache.so, which means it is not MySQL or memcache connections leaving last candidate as the only choice. I also could see some of the variables in GDB "bt" command output which also hinted what could be the problem.

By the way does anyone know any debugger which can connect to PHP process or apache with mod_php and provide backtrace in PHP terms not the one for zend engine ? That would be pretty cool.

Yet another great tool which you can use is server-status if you're running apache. This way you will see the URL which that process is processing and so get few more hints on what may be happening or even get repeatable example in some cases.

The tools I mentioned regarding figuring our what is happening with the process are not only helpful to debug sleeping connections with MySQL but many other cases when you see web application locking up or starting to runs in the tight loop consuming too much CPU time.

If you know any other tools which could be helpful in this regard would appreciate your comments. There might be some smarter tools out where for production tracing.

Entry posted by peter | 19 comments

Add to: | | | |