I start to see applications being built utilizing VIEWs functionality which appeared in MySQL 5.0 and quite frequently VIEWs are used to help in writing the queries – to keep queries simple without really thinking how it affects server performance.
Even worse than that – looking at the short table which just gets single row from the table by the key we think this is a simple query, while can be a real monster instead with complexity hidden away in VIEW definition.
Just the other day I worked on optimizing application which uses VIEWs and was looking at the long-running query which just joined 2 tables… I ran EXPLAIN for it and got 200 of rows in the result set just for explaining due to several layers of cascaded views built on top of one another so it is easy to write the queries, some of them it turn used subqueries subselects and derived tables.
It is also very dangerous if you assume MySQL would optimize your VIEWs same way as more advanced database systems would. Same as with subqueries and derived tables MySQL 5.0 will fail and perform very inefficiently in many counts.
MySQL has two ways of handling the VIEWS
Query merge, in which case VIEW is simply expanded as a macro or Temporary Table in which case VIEW is materialized to temporary tables (without indexes !) which is later used further in query execution.
There does not seem to be any optimizations applied to the query used for temporary table creation from the outer query and plus if you use more then one Temporary Tables views which you join together you may have serious issues because such tables do not get any indexes.
Let me now show a couple of examples.
Assume we have the comments table which holds users comments to the blog, naturally containing user_id which left comment, comment_id and comment text:
1 2 3 4 5 6 | CREATE TABLE `comments` ( `user_id` int(10) unsigned NOT NULL, `comment_id` int(10) unsigned NOT NULL, `message` text NOT NULL, PRIMARY KEY (`user_id`,`comment_id`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1 |
So how would you get a number of comments left by the given user?
1 2 3 4 5 6 7 | mysql> select count(*) from comments where user_id=5; +----------+ | count(*) | +----------+ | 1818 | +----------+ 1 row in set (0.00 sec) |
So how would we solve the same problem having things more modular and using MySQL VIEWs?
1 2 3 4 5 6 7 8 9 10 | mysql> create view user_counts as select user_id,count(*) cnt from comments group by user_id; Query OK, 0 rows affected (0.00 sec) mysql> select * from user_counts where user_id=5; +---------+------+ | user_id | cnt | +---------+------+ | 5 | 1818 | +---------+------+ 1 row in set (0.95 sec) |
So we create the view which gives us back counts for each user and can simply query from that table restricting by user_id.
If this would be handled properly inside MySQL there would be even good reason to do that – so later you can change your application and convert user_count to summary table avoid changing any queries directly. Unfortunately, it does not work.
It is interesting to see EXPLAIN for such query and time for the query which fetches everything from the VIEW – it is almost the same as getting only one row, and note even EXPLAIN takes the same amount of time:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | mysql> explain select * from user_counts where user_id=5 \G *************************** 1. row *************************** id: 1 select_type: PRIMARY table: type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 1001 Extra: Using where *************************** 2. row *************************** id: 2 select_type: DERIVED table: comments type: index possible_keys: NULL key: PRIMARY key_len: 8 ref: NULL rows: 1792695 Extra: Using index 2 rows in set (0.96 sec) mysql> select * from user_counts; +---------+------+ | user_id | cnt | +---------+------+ | 0 | 850 | | 1 | 1790 | | 2 | 1777 | | 3 | 1762 | | 4 | 1784 | .... | 999 | 1808 | | 1000 | 898 | +---------+------+ 1001 rows in set (0.96 sec) |
So now let’s create a very artificial query which will JOIN 2 views just to see how indexes are used:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | mysql> explain select uc.cnt+uc2.cnt from user_counts uc, user_counts uc2 where uc.user_id=uc2.user_id and uc.user_id=5 \G *************************** 1. row *************************** id: 1 select_type: PRIMARY table: type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 1001 Extra: Using where; Using join cache *************************** 2. row *************************** id: 1 select_type: PRIMARY table: type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 1001 Extra: Using where *************************** 3. row *************************** id: 3 select_type: DERIVED table: comments type: index possible_keys: NULL key: PRIMARY key_len: 8 ref: NULL rows: 1792695 Extra: Using index *************************** 4. row *************************** id: 2 select_type: DERIVED table: comments type: index possible_keys: NULL key: PRIMARY key_len: 8 ref: NULL rows: 1792695 Extra: Using index 4 rows in set (1.91 sec) |
As you can see we get 2 derived tables in which case are fully populated and “full join” used to join between them. In this particular case it is not that bad because “join cache” is used to perform it relatively efficient, however for large derived tables it will become a nightmare.
So be very careful implementing MySQL VIEWs in your application, especially ones which require temporary table execution method. VIEWs can be used with very small performance overhead but only in case, they are used with caution.
MySQL has long way to go getting queries with VIEWs properly optimized.
Hi Peter,
Good article as always. Your second code example has some text comments in it that don’t line-wrap and are hard to read.
I actually had previously utilized MySQL’s VIEWs in a project when they had just came out. I figured that they would actually optimize how other databases would however what I came to find was that it was one of the worst performance bottlenecks as my data set grew. What I came to find was that MySQL would join all of the data together first and then it would run through the search criteria.
We recently converted from Access to MySQL. In this transition, lot of queries were written as views in MySQL. The performance was so bad that I decided to rewrite all of them using the base tables directly. As you aptly said, MySQL does have a long way to go as far as views are concerned.
I think the problem is not the VIEWs themselves, but how one attempted to use it.
Putting an aggregate function into a VIEW seemed like a bad choice to begin with (especially one that forced a table scan for each use), and that bad initial choice is compounded if one plans on using it with any frequency. At that cost, you may as well have just written a trigger on the comments table to update a user_counts table (or re-generate it entirely using the query you used for your view) for each insert or delete. You could read the now summarized user_count table (that only regenerates when needed) with the benefits of an index to get a specific user’s posting count.
You do point out a good thing, and that is that people need to really think through what they are trying to ultimately get to and figure out a way to get that data as efficiently as possible. Just using a mechanism (VIEWs) simply because they are there and available to use does not make them necessarily a good choice. Also, don’t put queries into a VIEW that you would not deem efficient to run normally.
Chrisk,
It is both. MySQL Views could be optimized better. You can use MySQL Views wisely only in cases when they are optimized well.
Many people do not think about performance until it starts to hurt badly so they just write queries (and put them in the views) in a way it gets them info they want easiest way.
“Many people do not think about performance until it starts to hurt badly so they just write queries (and put them in the views) in a way it gets them info they want easiest way.”
That could be construed as a feature, not a bug. You know what they say about premature optimization.
Well… Not optimizing beyond the need is one thing. Thinking you never would need to optimize is completely different.
If I have a view used to join data (via a UNION select) stored in separate tables, then use a where clause when selecting from the view, is that a case where MySQL would be inefficient? I don’t want it to create a temporary table of ALL the data in ALL the tables, I really was hoping for more of a macro-expand kind of thing.
i see VIEWS like a good alternative to caching some of the queries. Example: you have a site, with a
categoriesmenu (parent->childs) that are conditioned to appear in the menu by the number ofproductsthey reffer to. Every time a user loads a page you load that menu. If you have a large database, you don’t want to execute each time that menu (query), so you only SELECT the view. i think this is a primary use of VIEWS besides caching reports that don’t contain indexes but full values. please correct me if i’m wrong.How would you use views to cache queries in MySQL ?
MySQL does not have materialized views so whenever you access view it will be always reevaluated.
I’ve been working with MySQL Views since release 5 was first made available.
It took me and my team about 2 months of testing to understand that it is just too soon to use Views.
What we did as a solution was to create an extra layer between PHP and MySQL (written in PHP). Hardcoding the complex queries in functions, with our own caching logic. This made it for us, for now.
Have MySQL views improved at all – for any version?
Not in 5.1 at least. 6.0 have some optimizations which would affect views but I’m not sure if there are general fixes.
Ascanio and others: Try using Stored Procedure, for this example.
DELIMITER $$
DROP PROCEDURE IF EXISTS
test2.user_count$$CREATE DEFINER=
root@localhostPROCEDUREuser_count(param1 int)BEGIN
SELECT count(*) FROM comments WHERE user_id=param1;
END $$
DELIMITER ;
Run the above code
then CALL user_count(5); It will run SELECT count(*) FROM comments WHERE user_id=5; and just as fast as the straight select (from my testing less than 0.001 milisecond difference, sometimes the CALL was faster, but thats just due to random flux you might notice a difference if you were making thousands of calls a second..
Views meant to hide the complexities in the user’s side. This helps you to shorten your query. Optimization is to be done from the MySQL server itself. But have you ever heard of something like indexing a view? Something else? As a developer, performance too matters.
This post makes it clear that indexes will not be used for views that use temporary tables. For views that do not use temporary tables, will ‘joining’ or ‘whereing’ on a column that is indexed in the underlying table use that index?
nice articel… thanks for your backmarking… 🙂
LOL, very useless article!
Author try to compare two uncompirable things!
Look here: Simple select is slower then view-select (ROFL):
==================================================
Simple select:
mysql> select count(*) from (select * from comments where user_id in (select user_id from comments where user_id = 5 group by user_id)) t2;
+———-+
| count(*) |
+———-+
| 100 |
+———-+
1 row in set (3.71 sec)
And from VIEW (CREATE VIEW user_counts AS SELECT * FROM comments):
mysql> select count(*) from user_counts where user_id = 5;
+———-+
| count(*) |
+———-+
| 100 |
+———-+
1 row in set (0.00 sec)
==================================================
Author’s problem NOT IN VIEW at all, but in using a view for non-view-based operations!
VIEW – mechanism to make a pseudo-tables for database users (for example, to make four different client lists based on three tables (account, client, client-address) tables: client’s names list, client’s address list, client’s detail info list. On this lists client-side-software maps grids, combo-boxes and other.
In author’s example, VIEW works slowly ONLY AND ONLY BECOUSE autor use a GROUP BY statement in VIEW!
PS: sorry for my english, i’m not from english-speaking country 🙂
Blakkky, you should learn the execution plan differences between joins and correlated subqueries.
Hmm, it appears that using views which could be a quicker way to make db calls especially in joins and subqueries… never realized how bad the performance is by using them. I wonder if this performance issue also occurs when creating triggers and stored procedures.
Barton Schwartz, plz read my post MINDFULLY! My example is a joke, that shows two uncompairable things. Author’s post is same as mine, he compare a GROUP BY SELECT and SIMPLE SELECT and say, that VIEW, based on GROUP BY SELECT is slowly than SIMPLE SELECT. Is it a right comparasion?
Again, author’s example is same as
SELECT cnt FROM (SELECT user_id,count(*) cnt FROM comments GROUP BY user_id) AS user_counts WHERE user_id = 5;
and he compare it with
SELECT count(*) FROM comments WHERE user_id=5;
In this case, first select (view) is SLOWLY, then simple select (it’s obvious, becouse “group by” with index filtering is slowly, that simple index filtering). Problem not in VIEW MECHANISM in mySQL at all, but in author’s approach to using view!
Correct test is something like this:
1st experiment:
mysql> CREATE VIEW user_counts AS SELECT user_id, comment_id, message FROM comments;
Query OK, 0 rows affected (0.02 sec)
mysql> SELECT user_id, count(*) FROM user_counts WHERE user_id = 5;
+———+———-+
| user_id | count(*) |
+———+———-+
| 5 | 100 |
+———+———-+
1 row in set (0.00 sec)
mysql> explain SELECT user_id, count(*) FROM user_counts WHERE user_id = 5;
+—-+————-+———-+——+—————+———+———+——-+——+————-+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+—-+————-+———-+——+—————+———+———+——-+——+————-+
| 1 | SIMPLE | comments | ref | PRIMARY | PRIMARY | 4 | const | 100 | Using index |
+—-+————-+———-+——+—————+———+———+——-+——+————-+
1 row in set (0.00 sec)
2nd experiment:
mysql> SELECT user_id, count(*) FROM comments WHERE user_id = 5;
+———+———-+
| user_id | count(*) |
+———+———-+
| 5 | 100 |
+———+———-+
1 row in set (0.00 sec)
mysql> explain SELECT user_id, count(*) FROM comments WHERE user_id = 5;
+—-+————-+———-+——+—————+———+———+——-+——+————-+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+—-+————-+———-+——+—————+———+———+——-+——+————-+
| 1 | SIMPLE | comments | ref | PRIMARY | PRIMARY | 4 | const | 100 | Using index |
+—-+————-+———-+——+—————+———+———+——-+——+————-+
1 row in set (0.00 sec)
In this case, this queries has EQUAL PERFORMANCE at EQUAL REQUESTS!
PS: I know, that in my test there is no need to use VIEW, but it’s a TEST, not production solution!
view is really need to make “interface pseudo-tables”, that uses from database-client software to make a database structure more flexible and make database-client applications not dependented from database real structure.
So if we have two or more large views that we would like to join are we pretty much stuck? Or is there such a thing as an indexed view?
Tiberius,
It is possible. Devil is as usually in details – MySQL have different ways of executions queries with views – if it just “merges” the query it may perform well if you get 2 large temporary tables which need to be joined on the final stage it will be rather slow.
EXPLAIN should tell you that.
@Peter
Thank you, I just did an “explain” on the query and it revealed some very interesting details!
Could you perhaps rephrase this? I’m confused …
“if it just “merges†the query it may perform well if you get 2 large temporary tables which need to be joined on the final stage it will be rather slow.”
I’m curious what steps / info we can refer Tiberius to for optimizing his query. I have a view in a query of my own and its speed is turtle-fast (sarcasm, hehe). The EXPLAIN convinces me that temporary tables are being built and then joined.
However, since even when I copy my view data into a real table before doing the joins, adding the appropriate indexes, and still get horrible performance – it probably is more MySQL’s join that is failing me, rather than the view.
Tiberius,
MySQL has 2 ways to deal with views – MERGE and TMPTABLE, see http://dev.mysql.com/doc/refman/5.0/en/create-view.html
Looking at EXPLAIN you should see either all views collapsed and query looking as a simple join or you should see temporary table and probably 2 rows with “ALL” access type corresponding to doing full join on temporary table.
Note if MySQL does “Merge” for view it does not matter it is able to execute join efficiently in all cases, but this is other topic.
First of all, thank you both for working with me on this, and thanks for the heads up on the MERGE and TEMPTABLE algorithms. I’ll try creating the views with both algorithms and let you know how it goes.
Basically, I’m in the process of importing the content of a pre-existing mysql db into Drupal, but the table and field names change when you do that. I’m looking to essentially create “aliases” from the new (Drupal) table and field names to the old ones, so that our HUGE perl library can continue to function without us rewriting every single query. So I thought views would be a good solution. I have yet to see if 5 or 6 views can be joined without a huge performance hit …
Tiberius: You’re going to lose the ability to reliably write to the db from whatever application is using the views. MySQL views can be used for updates but its application is limited in many ways, you’d need to test all update scenarios as well.
SOLVED! It turns out that I just needed to add indices. The column names in the ‘where’ clause of the slow query were not indices, so the query was running really slowly. Once I added the appropriate indices to the tables, THEN created the views everything worked! So it turns out that views were not the culprit at all, it was querying on “un-indexed” columns that was the problem.
mysql view are good, but certain care is required –
1. tables on which you want to create views, should have proper primary key and indexes (make sure you do it before dumping data in the tables)
2. skip any where clause, or group by clause
3. drop off any columns which you dont require, or create multiple views, based on the column required
4. increase sql cache / memory (hosting providers … ahem ahem!)
5. do not mix views with outer joins
Depending on how often you are updating the data, it can be a lot more efficient to use a stored procedure to generate a table – effectively a materialized view – rather than using an actual MySQL.
In my application I have several hierarchies – customer, product, division. New products are added several times a day; new customers/divisions, much more rarely. If materialized views were supported, I would use that feature to de-normalize the static data so the frontend doesn’t need to generate a complex query to get the results it needs.
Using a MySQL view was an option I considered, but quickly rejected on the basis that the server would generally re-evaluate the underlying query each time the view is queried.
Because the write/query ratio on the static data is very low in my application, I have instead used stored procedures to drop and recreate the tables with the de-normalized data. This process takes a minute or two to perform (the normal tables are quite large), but given the rate of writes this is much more efficient than using even an optimized view.
I just tried views, and indeed it was slow (overall for auto completion purposes).
instead of “select * form my_view where my_field = $prm”, I create a temporary table at the start of the process “create temporary table my_temp as (select * from my_view)”, and do the selects on the temp table.
Yet, mysql has a long walk to go in this domain compared to competition.
Roselan can you plz explain to me how you created the temp table as i dont know how to do it. Tried also to use view but that was really really slow so i cant use it. thanks for helping greetings Raka
very nice article, well as per my understanding Mysql views can be used for many reasons like to simplify complex queries and giving limited access data to the specific users. etc, I agree that there are few issues with it as long as performance is concerned but it also has great power.
i still got the same problem as mentioned before maybe someone can tell me how to solve it as i dont get it working. Would be nice if someone got mail me the answer or answer me in here. Thanks in advance
hmmmm i tried to use the sql code mentioned above but somehow i got an error in line 27 and i dont know where i did go wrong. i just copied it is the an error inside which i dont seE?
The code did work out fine when I have tried it…Maybe you have used the wrong code!Anyways great blog. Keep up the good work.
when i tried the code piece the last time it worked without any problems. now i wanted to use it on another project of mine but suddenly its seems not to work. but i have to say that i use a different server with other software now maybe there are some issues i hope i*’ll figure it out soon.
i didnt encounter any problems when i was using the code what kind of server system do you use?
The complete syntax of the CREATE VIEW command looks like this:
CREATE [OR REPLACE] [ALGORITHM = UNDEFINED | MERGE | TEMPTABLE]
VIEW name [(columnlist)] AS select command
[WITH [CASCADED | LOCAL] CHECK OPTION]
ALGORITHM takes three values: MERGE, TEMPTABLE, or UNDEFINED.
By default, MySQL always uses UNDEFINED, if no ALGORITHM clause is present (can be determined via SHOW CREATE TABLE viewname).
Mysql prefers MERGE over TEMPTABLE if possible, because MERGE is usually more efficient.
// ***Views have slow performance on tables with large amount of data
http://bugs.mysql.com/bug.php?id=23135
To Raka: I didn’t face any problem…what type of server did you use? what configuration? Probably the reason is in config files…
i get an error if i use views for my database but i guess its just because it is very big and my server is too slow have to move to new one. be careful with big data amounts
i trie the code from shelley but somehow i get an error when i try to use the Merge in my database. i already tried it several times but i cant find the error. are you sure that the code is ok?
thanks for the post .. is helpfull .. also at shelley .. i have this needed for my work.
The code works 100% correct!! I don’t know why everybody got that problems…
The complete syntax of the CREATE VIEW command looks like this:
CREATE [OR REPLACE] [ALGORITHM = UNDEFINED | MERGE | TEMPTABLE]
VIEW name [(columnlist)] AS select command
[WITH [CASCADED | LOCAL] CHECK OPTION]
ALGORITHM takes three values: MERGE, TEMPTABLE, or UNDEFINED.
By default, MySQL always uses UNDEFINED, if no ALGORITHM clause is present (can be determined via SHOW CREATE TABLE viewname).
Mysql prefers MERGE over TEMPTABLE if possible, because MERGE is usually more efficient.
// ***Views have slow performance on tables with large amount of data
http://bugs.mysql.com/bug.php?id=23135
My experience is that troubles with MySql always come up when there are peaks of traffic. It’s for those peaks that you really have to optimize everything. Thanks for good advice, I’ll keep it in mind.
Great article – was really trying to decide if views were worth using as I’ve noticed the performance hit.
My database isn’t so large and I’m the only one using it so I think I’ll just skip Views.
Might write PHP functions so I can easily re-run complex queries.
Would rather a fast complex query then a slow (but simpler to use) View.
Is this still true for MySQL 5.5?
“Expanded as a macro or creating temporary tables”
When would it benefit a view to create a temporary tables over expanding as a macro?
And can one dictate the view to force it to run as a macro then to use the unindexed temporary table.
Truly I thought (albit incorrectly) that views were equal or better than the select statement they were based off.
But lo and behold to my surprise MySQL made it quite evident this is not that case.
In my situation I had 5 tables left outer joining. (about 2 of those tables were in the thousands, one was in the tens of thousands). The view worked fine.
Joined one more table and all hell broke loose! The select statement was almightily indexed for instant execution. The view however, did absolutely nothing for the longest amount of time after running a simple select off it.
Dropped the view and recreated it. Recreated it as some other name. Restarted the db server. The view did nothing. Removed the newly joined table and created the view again. The view was meant to never run again. Sad really.
Created a table off the select statement. Got a view to read from some simple wheres and voila!
BUT, however, i miss the old view immensely and here I am hoping to get it back and running.
Thanks! Good article m8!