MySQL CREATE INDEX slowing down
My analytics table has 190M rows, about 150GB ins size.
Storage engine is MyISAM with
key-buffer-size=16G
myisam_sort_buffer_size=2G
myisam-max-sort-file-size=200G.
Machine has 32GB memory.
While creating 10 simple indexes (no covering, no prefix, just column names using BTREE) I noticed:
- first index - 40 minutes
- second index - 50 minutes
- ...
- fifth index - 2 hours
- eleventh index - 3 1/2 hours
Each index adds between 1GB and 3GB to the .myi file.
Has anybody experienced a similar increase in run-time? Is there any way to get the performance of the first index creation for the other indexes?
mysql performance index myisam
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
My analytics table has 190M rows, about 150GB ins size.
Storage engine is MyISAM with
key-buffer-size=16G
myisam_sort_buffer_size=2G
myisam-max-sort-file-size=200G.
Machine has 32GB memory.
While creating 10 simple indexes (no covering, no prefix, just column names using BTREE) I noticed:
- first index - 40 minutes
- second index - 50 minutes
- ...
- fifth index - 2 hours
- eleventh index - 3 1/2 hours
Each index adds between 1GB and 3GB to the .myi file.
Has anybody experienced a similar increase in run-time? Is there any way to get the performance of the first index creation for the other indexes?
mysql performance index myisam
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.
– mootmoot
Jun 14 '16 at 14:41
add a comment |
My analytics table has 190M rows, about 150GB ins size.
Storage engine is MyISAM with
key-buffer-size=16G
myisam_sort_buffer_size=2G
myisam-max-sort-file-size=200G.
Machine has 32GB memory.
While creating 10 simple indexes (no covering, no prefix, just column names using BTREE) I noticed:
- first index - 40 minutes
- second index - 50 minutes
- ...
- fifth index - 2 hours
- eleventh index - 3 1/2 hours
Each index adds between 1GB and 3GB to the .myi file.
Has anybody experienced a similar increase in run-time? Is there any way to get the performance of the first index creation for the other indexes?
mysql performance index myisam
My analytics table has 190M rows, about 150GB ins size.
Storage engine is MyISAM with
key-buffer-size=16G
myisam_sort_buffer_size=2G
myisam-max-sort-file-size=200G.
Machine has 32GB memory.
While creating 10 simple indexes (no covering, no prefix, just column names using BTREE) I noticed:
- first index - 40 minutes
- second index - 50 minutes
- ...
- fifth index - 2 hours
- eleventh index - 3 1/2 hours
Each index adds between 1GB and 3GB to the .myi file.
Has anybody experienced a similar increase in run-time? Is there any way to get the performance of the first index creation for the other indexes?
mysql performance index myisam
mysql performance index myisam
edited Jun 14 '16 at 15:01
RolandoMySQLDBA
143k24226383
143k24226383
asked Jun 14 '16 at 14:27
Wolf MetznerWolf Metzner
61
61
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 9 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.
– mootmoot
Jun 14 '16 at 14:41
add a comment |
Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.
– mootmoot
Jun 14 '16 at 14:41
Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.
– mootmoot
Jun 14 '16 at 14:41
Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.
– mootmoot
Jun 14 '16 at 14:41
add a comment |
2 Answers
2
active
oldest
votes
You must be doing something odd with your DDL or you have a very, very old version of MySQL.
MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.
Oct 10, 2006
: Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)
May 12, 2011
: Does MySQL still handle indexes in this way? (where I mention this again)
Without any additional details from you, I can only guess. Here it goes:
I suspect your must be just doing the following:
ALTER TABLE mytable ADD INDEX (column01);
ALTER TABLE mytable ADD INDEX (column02);
ALTER TABLE mytable ADD INDEX (column03);
...
ALTER TABLE mytable ADD INDEX (column10);
ALTER TABLE mytable ADD INDEX (column11);
I would like to suggest the following
SUGGESTION #1 : Build all 11 indexes at the same time
You should create all 11 indexes in a single DDL statement
ALTER TABLE myisamtable
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
SUGGESTION #2 : Delay indexes building
You can disable the building of non-unique indexes so that sorting and be done.
Doing so can make a smaller BTREE for each index
ALTER TABLE mytable DISABLE KEYS;
...
ALTER TABLE mytable ENABLE KEYS;
SUGGESTION #3 : Use a temp table
If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.
Here is a sample of what I just described to you
CREATE TABLE mytable_new LIKE mytable;
ALTER TABLE mytable_new
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
ALTER TABLE mytable_new DISABLE KEYS;
INSERT INTO mytable_new SELECT * FROM mytable;
ALTER TABLE mytable_new ENABLE KEYS;
ALTER TABLE mytable RENAME mytable_old;
ALTER TABLE mytable_new RENAME mytable;
DROP TABLE mytable_old;
GIVE IT A TRY !!!
If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.
– RolandoMySQLDBA
Jun 15 '16 at 15:41
add a comment |
Rolando,
Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
My statements were:
CREATE INDEX idx1 ....;
CREATE INDEX idx2 ....;
CREATE INDEX idx3 ....;
...
and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use
ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...
in the future.
Thanks! Problem solved .. question answered!
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "182"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f141192%2fmysql-create-index-slowing-down%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
You must be doing something odd with your DDL or you have a very, very old version of MySQL.
MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.
Oct 10, 2006
: Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)
May 12, 2011
: Does MySQL still handle indexes in this way? (where I mention this again)
Without any additional details from you, I can only guess. Here it goes:
I suspect your must be just doing the following:
ALTER TABLE mytable ADD INDEX (column01);
ALTER TABLE mytable ADD INDEX (column02);
ALTER TABLE mytable ADD INDEX (column03);
...
ALTER TABLE mytable ADD INDEX (column10);
ALTER TABLE mytable ADD INDEX (column11);
I would like to suggest the following
SUGGESTION #1 : Build all 11 indexes at the same time
You should create all 11 indexes in a single DDL statement
ALTER TABLE myisamtable
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
SUGGESTION #2 : Delay indexes building
You can disable the building of non-unique indexes so that sorting and be done.
Doing so can make a smaller BTREE for each index
ALTER TABLE mytable DISABLE KEYS;
...
ALTER TABLE mytable ENABLE KEYS;
SUGGESTION #3 : Use a temp table
If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.
Here is a sample of what I just described to you
CREATE TABLE mytable_new LIKE mytable;
ALTER TABLE mytable_new
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
ALTER TABLE mytable_new DISABLE KEYS;
INSERT INTO mytable_new SELECT * FROM mytable;
ALTER TABLE mytable_new ENABLE KEYS;
ALTER TABLE mytable RENAME mytable_old;
ALTER TABLE mytable_new RENAME mytable;
DROP TABLE mytable_old;
GIVE IT A TRY !!!
If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.
– RolandoMySQLDBA
Jun 15 '16 at 15:41
add a comment |
You must be doing something odd with your DDL or you have a very, very old version of MySQL.
MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.
Oct 10, 2006
: Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)
May 12, 2011
: Does MySQL still handle indexes in this way? (where I mention this again)
Without any additional details from you, I can only guess. Here it goes:
I suspect your must be just doing the following:
ALTER TABLE mytable ADD INDEX (column01);
ALTER TABLE mytable ADD INDEX (column02);
ALTER TABLE mytable ADD INDEX (column03);
...
ALTER TABLE mytable ADD INDEX (column10);
ALTER TABLE mytable ADD INDEX (column11);
I would like to suggest the following
SUGGESTION #1 : Build all 11 indexes at the same time
You should create all 11 indexes in a single DDL statement
ALTER TABLE myisamtable
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
SUGGESTION #2 : Delay indexes building
You can disable the building of non-unique indexes so that sorting and be done.
Doing so can make a smaller BTREE for each index
ALTER TABLE mytable DISABLE KEYS;
...
ALTER TABLE mytable ENABLE KEYS;
SUGGESTION #3 : Use a temp table
If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.
Here is a sample of what I just described to you
CREATE TABLE mytable_new LIKE mytable;
ALTER TABLE mytable_new
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
ALTER TABLE mytable_new DISABLE KEYS;
INSERT INTO mytable_new SELECT * FROM mytable;
ALTER TABLE mytable_new ENABLE KEYS;
ALTER TABLE mytable RENAME mytable_old;
ALTER TABLE mytable_new RENAME mytable;
DROP TABLE mytable_old;
GIVE IT A TRY !!!
If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.
– RolandoMySQLDBA
Jun 15 '16 at 15:41
add a comment |
You must be doing something odd with your DDL or you have a very, very old version of MySQL.
MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.
Oct 10, 2006
: Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)
May 12, 2011
: Does MySQL still handle indexes in this way? (where I mention this again)
Without any additional details from you, I can only guess. Here it goes:
I suspect your must be just doing the following:
ALTER TABLE mytable ADD INDEX (column01);
ALTER TABLE mytable ADD INDEX (column02);
ALTER TABLE mytable ADD INDEX (column03);
...
ALTER TABLE mytable ADD INDEX (column10);
ALTER TABLE mytable ADD INDEX (column11);
I would like to suggest the following
SUGGESTION #1 : Build all 11 indexes at the same time
You should create all 11 indexes in a single DDL statement
ALTER TABLE myisamtable
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
SUGGESTION #2 : Delay indexes building
You can disable the building of non-unique indexes so that sorting and be done.
Doing so can make a smaller BTREE for each index
ALTER TABLE mytable DISABLE KEYS;
...
ALTER TABLE mytable ENABLE KEYS;
SUGGESTION #3 : Use a temp table
If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.
Here is a sample of what I just described to you
CREATE TABLE mytable_new LIKE mytable;
ALTER TABLE mytable_new
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
ALTER TABLE mytable_new DISABLE KEYS;
INSERT INTO mytable_new SELECT * FROM mytable;
ALTER TABLE mytable_new ENABLE KEYS;
ALTER TABLE mytable RENAME mytable_old;
ALTER TABLE mytable_new RENAME mytable;
DROP TABLE mytable_old;
GIVE IT A TRY !!!
You must be doing something odd with your DDL or you have a very, very old version of MySQL.
MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.
Oct 10, 2006
: Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)
May 12, 2011
: Does MySQL still handle indexes in this way? (where I mention this again)
Without any additional details from you, I can only guess. Here it goes:
I suspect your must be just doing the following:
ALTER TABLE mytable ADD INDEX (column01);
ALTER TABLE mytable ADD INDEX (column02);
ALTER TABLE mytable ADD INDEX (column03);
...
ALTER TABLE mytable ADD INDEX (column10);
ALTER TABLE mytable ADD INDEX (column11);
I would like to suggest the following
SUGGESTION #1 : Build all 11 indexes at the same time
You should create all 11 indexes in a single DDL statement
ALTER TABLE myisamtable
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
SUGGESTION #2 : Delay indexes building
You can disable the building of non-unique indexes so that sorting and be done.
Doing so can make a smaller BTREE for each index
ALTER TABLE mytable DISABLE KEYS;
...
ALTER TABLE mytable ENABLE KEYS;
SUGGESTION #3 : Use a temp table
If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.
Here is a sample of what I just described to you
CREATE TABLE mytable_new LIKE mytable;
ALTER TABLE mytable_new
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
ALTER TABLE mytable_new DISABLE KEYS;
INSERT INTO mytable_new SELECT * FROM mytable;
ALTER TABLE mytable_new ENABLE KEYS;
ALTER TABLE mytable RENAME mytable_old;
ALTER TABLE mytable_new RENAME mytable;
DROP TABLE mytable_old;
GIVE IT A TRY !!!
edited Apr 13 '17 at 12:42
Community♦
1
1
answered Jun 14 '16 at 15:00
RolandoMySQLDBARolandoMySQLDBA
143k24226383
143k24226383
If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.
– RolandoMySQLDBA
Jun 15 '16 at 15:41
add a comment |
If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.
– RolandoMySQLDBA
Jun 15 '16 at 15:41
If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.
– RolandoMySQLDBA
Jun 15 '16 at 15:41
If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.
– RolandoMySQLDBA
Jun 15 '16 at 15:41
add a comment |
Rolando,
Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
My statements were:
CREATE INDEX idx1 ....;
CREATE INDEX idx2 ....;
CREATE INDEX idx3 ....;
...
and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use
ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...
in the future.
Thanks! Problem solved .. question answered!
add a comment |
Rolando,
Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
My statements were:
CREATE INDEX idx1 ....;
CREATE INDEX idx2 ....;
CREATE INDEX idx3 ....;
...
and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use
ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...
in the future.
Thanks! Problem solved .. question answered!
add a comment |
Rolando,
Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
My statements were:
CREATE INDEX idx1 ....;
CREATE INDEX idx2 ....;
CREATE INDEX idx3 ....;
...
and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use
ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...
in the future.
Thanks! Problem solved .. question answered!
Rolando,
Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
My statements were:
CREATE INDEX idx1 ....;
CREATE INDEX idx2 ....;
CREATE INDEX idx3 ....;
...
and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use
ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...
in the future.
Thanks! Problem solved .. question answered!
answered Jun 15 '16 at 15:38
Wolf MetznerWolf Metzner
61
61
add a comment |
add a comment |
Thanks for contributing an answer to Database Administrators Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f141192%2fmysql-create-index-slowing-down%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.
– mootmoot
Jun 14 '16 at 14:41