I have one question regarding your comment

“Sort by column in leading table if you have JOIN with ORDER BY … LIMIT you should try hard to have sorting column(s) to be in the leading table. If ORDER BY is going by field from the table which is not first in the join order index can’t be used.”

I have a similar situation where I can not have sorting columns in leading table; in such case please suggest any work around.

Thanks in anticipation.
Rbk

I am going through your article its found pretty much inserting. I need one help as i am displaying my property query which is displaying properly and ordering the result by following 4 column

p.is_move_up DESC,REPLACE(IFNULL(p.is_feature,”"),”2015-12-31″,”") DESC,
p.listing_order DESC,p.propid DESC

In order site, properties are been displayed in this order: Moveup paid, Paid featured, paid, trial then expired. However, we want to show within each group the properties random and make them rotate. Now,
the properties do not rotate so it is not fair for users to see always the same properties in same order. So, we need the rotation to be within same group not mixing all.

Query :-

SELECT DISTINCT(p.propid),p.category_id,p.people_sleep,p.bedrooms,p.bathrooms,p.children,
p.airport,p.features,p.pets,p.smoke, p.wheelchair,p.prop_title,p.property_type,prop_date,
display_calender,other_url_calender,is_feature
FROM property p
LEFT JOIN category ON (p.category_id=category.id)
WHERE p.active IN(“yes”,”d”) AND p.is_feature >= SYSDATE()
ORDER BY p.is_move_up DESC,REPLACE(IFNULL(p.is_feature,”"),”2015-12-31″,”") DESC,
p.listing_order DESC,p.propid DESC

Can you please help me into this.

This is a hard one especially if you add ranges as MySQL will not be able to both use range for selection and sort by different column. You may and up with large scans and sorts which can use many resources. Sphinx (http://www.sphinxsearch.com) is often a good alternative in such cases even if you’re not using full text search query

I’ve been reading your articles for some time now, especially since I’ve been running into issues with wrapping my head around this whole MySQL optimization stuff.

Anyways, I get the jist of it, however, I run a real estate website (displays listings, etc, in a search results page), and there are just too many variables to try and cover an index on each scenario… I’d end up with 20 indexes, and I fear that any update/insert/delete queries to that table would blow up the internet.

I’m not looking for anybody to write me anything, I’m just looking to get a better understanding of this:

SELECT
`id`,
… more *necessary* columns …
FROM
`listings`
WHERE
`category` = 1
AND `country` = 1
AND `state` = 45
AND `city` = 35859
AND `visible` = 1
ORDER BY
`price` ASC
LIMIT 0, 10

Now, `category` numbers represent different listing types, ie. 1 = For Sale, 2 = For Rent, 3 = Pending Sale, and so on. 1 (For Sale is by the far the most popular, representing at least 75% of all listings in the table; probably more).

`country`, `state`, `city` are always in the query. `visible` is either a 1 or 0. 1 meaning the listing has not expired, and 0 meaning it has. Probably 85%+ instances of `visible` = 1 in the table.

Now, there are also other WHERE conditions that are possible, like WHERE `price` = 300000, or `bedrooms` = 2, or `bathrooms` = 3, etc. And who knows what the user is going to enter in.

Then, there is the order by on top of that.

What I have been looking for is how to accommodate these kinds of variables when there are so many possible scenarios? When I do an ORDER BY `price` DESC, it can take 4-5 seconds, maybe more… I guess that’s not the end of the world, but our traffic is always growing, and that will end up being unacceptable.

Any thoughts?

- Batman

We have a tool which is processing 30 million records, this tool is a migration project where we are using perl script to do that

we need to compare two tables where we are doing like this

for using counter of 50,000 records in chunk size

and then we are using

select column1, column2 from table order by table limit index_num(21 million), (500,000)

initially for smaller dataset, things are working fine (26 secs to 50 secs)

when index_num is growing, it is taking almost 30 mins to do that

We want to optimize this one.

As per your recent posts,

we need to have index, for column_1, we are creating indexes ..still it is not optimal.

Thanks for the quick response.

I have removed the index on category_id. After removing it Explain for the below select query is as follows:

EXPLAIN select newsid from news where category_id=53 and approval=’yes’ order by date_created desc limit 0,10

id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE news index NULL date_created 9 NULL 10 Using where

This is fine i guess.
Another question is: If there are 56 records for category_id=53 and i want to display 100 records for all given categories in news table, why does below query examine 100 records for category_id=53 when there are only 56 of them?

EXPLAIN select newsid from news where category_id=53 and approval=’yes’ order by date_created desc limit 0,100

id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE news index NULL date_created 9 NULL 100 Using where

Most of the rows are displayed based on category_id, so I want to know if I am doing it correctly.

If this is confusing I could send the table structures and queries.

In this post you have mentioned about DOS attack, do you mean paging records can cause DOS attack ?

Any help would be highly appreciated.

you seems to have poor index setup to begin with. You have “index scan” while you should have ref if you have (category_id,approval,date_created) for all queries.
If Index is scanned the more rows have to be scanned to provide 10 results the more selective where clause is.