MySQL CREATE INDEX slowing down












1















My analytics table has 190M rows, about 150GB ins size.
Storage engine is MyISAM with



key-buffer-size=16G
myisam_sort_buffer_size=2G
myisam-max-sort-file-size=200G.


Machine has 32GB memory.
While creating 10 simple indexes (no covering, no prefix, just column names using BTREE) I noticed:




  • first index - 40 minutes

  • second index - 50 minutes

  • ...

  • fifth index - 2 hours

  • eleventh index - 3 1/2 hours


Each index adds between 1GB and 3GB to the .myi file.
Has anybody experienced a similar increase in run-time? Is there any way to get the performance of the first index creation for the other indexes?










share|improve this question
















bumped to the homepage by Community 9 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
















  • Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.

    – mootmoot
    Jun 14 '16 at 14:41
















1















My analytics table has 190M rows, about 150GB ins size.
Storage engine is MyISAM with



key-buffer-size=16G
myisam_sort_buffer_size=2G
myisam-max-sort-file-size=200G.


Machine has 32GB memory.
While creating 10 simple indexes (no covering, no prefix, just column names using BTREE) I noticed:




  • first index - 40 minutes

  • second index - 50 minutes

  • ...

  • fifth index - 2 hours

  • eleventh index - 3 1/2 hours


Each index adds between 1GB and 3GB to the .myi file.
Has anybody experienced a similar increase in run-time? Is there any way to get the performance of the first index creation for the other indexes?










share|improve this question
















bumped to the homepage by Community 9 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
















  • Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.

    – mootmoot
    Jun 14 '16 at 14:41














1












1








1








My analytics table has 190M rows, about 150GB ins size.
Storage engine is MyISAM with



key-buffer-size=16G
myisam_sort_buffer_size=2G
myisam-max-sort-file-size=200G.


Machine has 32GB memory.
While creating 10 simple indexes (no covering, no prefix, just column names using BTREE) I noticed:




  • first index - 40 minutes

  • second index - 50 minutes

  • ...

  • fifth index - 2 hours

  • eleventh index - 3 1/2 hours


Each index adds between 1GB and 3GB to the .myi file.
Has anybody experienced a similar increase in run-time? Is there any way to get the performance of the first index creation for the other indexes?










share|improve this question
















My analytics table has 190M rows, about 150GB ins size.
Storage engine is MyISAM with



key-buffer-size=16G
myisam_sort_buffer_size=2G
myisam-max-sort-file-size=200G.


Machine has 32GB memory.
While creating 10 simple indexes (no covering, no prefix, just column names using BTREE) I noticed:




  • first index - 40 minutes

  • second index - 50 minutes

  • ...

  • fifth index - 2 hours

  • eleventh index - 3 1/2 hours


Each index adds between 1GB and 3GB to the .myi file.
Has anybody experienced a similar increase in run-time? Is there any way to get the performance of the first index creation for the other indexes?







mysql performance index myisam






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Jun 14 '16 at 15:01









RolandoMySQLDBA

143k24226383




143k24226383










asked Jun 14 '16 at 14:27









Wolf MetznerWolf Metzner

61




61





bumped to the homepage by Community 9 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.







bumped to the homepage by Community 9 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.















  • Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.

    – mootmoot
    Jun 14 '16 at 14:41



















  • Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.

    – mootmoot
    Jun 14 '16 at 14:41

















Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.

– mootmoot
Jun 14 '16 at 14:41





Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.

– mootmoot
Jun 14 '16 at 14:41










2 Answers
2






active

oldest

votes


















0














You must be doing something odd with your DDL or you have a very, very old version of MySQL.



MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.





  • Oct 10, 2006 : Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)


  • May 12, 2011 : Does MySQL still handle indexes in this way? (where I mention this again)


Without any additional details from you, I can only guess. Here it goes:



I suspect your must be just doing the following:



ALTER TABLE mytable ADD INDEX (column01);
ALTER TABLE mytable ADD INDEX (column02);
ALTER TABLE mytable ADD INDEX (column03);
...
ALTER TABLE mytable ADD INDEX (column10);
ALTER TABLE mytable ADD INDEX (column11);


I would like to suggest the following



SUGGESTION #1 : Build all 11 indexes at the same time



You should create all 11 indexes in a single DDL statement



ALTER TABLE myisamtable
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;


SUGGESTION #2 : Delay indexes building



You can disable the building of non-unique indexes so that sorting and be done.



Doing so can make a smaller BTREE for each index



ALTER TABLE mytable DISABLE KEYS;
...
ALTER TABLE mytable ENABLE KEYS;


SUGGESTION #3 : Use a temp table



If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.



Here is a sample of what I just described to you



CREATE TABLE mytable_new LIKE mytable;
ALTER TABLE mytable_new
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
ALTER TABLE mytable_new DISABLE KEYS;
INSERT INTO mytable_new SELECT * FROM mytable;
ALTER TABLE mytable_new ENABLE KEYS;
ALTER TABLE mytable RENAME mytable_old;
ALTER TABLE mytable_new RENAME mytable;
DROP TABLE mytable_old;


GIVE IT A TRY !!!






share|improve this answer


























  • If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.

    – RolandoMySQLDBA
    Jun 15 '16 at 15:41



















0














Rolando,
Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
My statements were:



CREATE INDEX idx1 ....;
CREATE INDEX idx2 ....;
CREATE INDEX idx3 ....;
...


and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use



ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...


in the future.



Thanks! Problem solved .. question answered!






share|improve this answer























    Your Answer








    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "182"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f141192%2fmysql-create-index-slowing-down%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    You must be doing something odd with your DDL or you have a very, very old version of MySQL.



    MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.





    • Oct 10, 2006 : Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)


    • May 12, 2011 : Does MySQL still handle indexes in this way? (where I mention this again)


    Without any additional details from you, I can only guess. Here it goes:



    I suspect your must be just doing the following:



    ALTER TABLE mytable ADD INDEX (column01);
    ALTER TABLE mytable ADD INDEX (column02);
    ALTER TABLE mytable ADD INDEX (column03);
    ...
    ALTER TABLE mytable ADD INDEX (column10);
    ALTER TABLE mytable ADD INDEX (column11);


    I would like to suggest the following



    SUGGESTION #1 : Build all 11 indexes at the same time



    You should create all 11 indexes in a single DDL statement



    ALTER TABLE myisamtable
    ADD INDEX (column01),
    ADD INDEX (column02),
    ADD INDEX (column03),
    ...
    ADD INDEX (column10),
    ADD INDEX (column11)
    ;


    SUGGESTION #2 : Delay indexes building



    You can disable the building of non-unique indexes so that sorting and be done.



    Doing so can make a smaller BTREE for each index



    ALTER TABLE mytable DISABLE KEYS;
    ...
    ALTER TABLE mytable ENABLE KEYS;


    SUGGESTION #3 : Use a temp table



    If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.



    Here is a sample of what I just described to you



    CREATE TABLE mytable_new LIKE mytable;
    ALTER TABLE mytable_new
    ADD INDEX (column01),
    ADD INDEX (column02),
    ADD INDEX (column03),
    ...
    ADD INDEX (column10),
    ADD INDEX (column11)
    ;
    ALTER TABLE mytable_new DISABLE KEYS;
    INSERT INTO mytable_new SELECT * FROM mytable;
    ALTER TABLE mytable_new ENABLE KEYS;
    ALTER TABLE mytable RENAME mytable_old;
    ALTER TABLE mytable_new RENAME mytable;
    DROP TABLE mytable_old;


    GIVE IT A TRY !!!






    share|improve this answer


























    • If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.

      – RolandoMySQLDBA
      Jun 15 '16 at 15:41
















    0














    You must be doing something odd with your DDL or you have a very, very old version of MySQL.



    MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.





    • Oct 10, 2006 : Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)


    • May 12, 2011 : Does MySQL still handle indexes in this way? (where I mention this again)


    Without any additional details from you, I can only guess. Here it goes:



    I suspect your must be just doing the following:



    ALTER TABLE mytable ADD INDEX (column01);
    ALTER TABLE mytable ADD INDEX (column02);
    ALTER TABLE mytable ADD INDEX (column03);
    ...
    ALTER TABLE mytable ADD INDEX (column10);
    ALTER TABLE mytable ADD INDEX (column11);


    I would like to suggest the following



    SUGGESTION #1 : Build all 11 indexes at the same time



    You should create all 11 indexes in a single DDL statement



    ALTER TABLE myisamtable
    ADD INDEX (column01),
    ADD INDEX (column02),
    ADD INDEX (column03),
    ...
    ADD INDEX (column10),
    ADD INDEX (column11)
    ;


    SUGGESTION #2 : Delay indexes building



    You can disable the building of non-unique indexes so that sorting and be done.



    Doing so can make a smaller BTREE for each index



    ALTER TABLE mytable DISABLE KEYS;
    ...
    ALTER TABLE mytable ENABLE KEYS;


    SUGGESTION #3 : Use a temp table



    If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.



    Here is a sample of what I just described to you



    CREATE TABLE mytable_new LIKE mytable;
    ALTER TABLE mytable_new
    ADD INDEX (column01),
    ADD INDEX (column02),
    ADD INDEX (column03),
    ...
    ADD INDEX (column10),
    ADD INDEX (column11)
    ;
    ALTER TABLE mytable_new DISABLE KEYS;
    INSERT INTO mytable_new SELECT * FROM mytable;
    ALTER TABLE mytable_new ENABLE KEYS;
    ALTER TABLE mytable RENAME mytable_old;
    ALTER TABLE mytable_new RENAME mytable;
    DROP TABLE mytable_old;


    GIVE IT A TRY !!!






    share|improve this answer


























    • If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.

      – RolandoMySQLDBA
      Jun 15 '16 at 15:41














    0












    0








    0







    You must be doing something odd with your DDL or you have a very, very old version of MySQL.



    MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.





    • Oct 10, 2006 : Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)


    • May 12, 2011 : Does MySQL still handle indexes in this way? (where I mention this again)


    Without any additional details from you, I can only guess. Here it goes:



    I suspect your must be just doing the following:



    ALTER TABLE mytable ADD INDEX (column01);
    ALTER TABLE mytable ADD INDEX (column02);
    ALTER TABLE mytable ADD INDEX (column03);
    ...
    ALTER TABLE mytable ADD INDEX (column10);
    ALTER TABLE mytable ADD INDEX (column11);


    I would like to suggest the following



    SUGGESTION #1 : Build all 11 indexes at the same time



    You should create all 11 indexes in a single DDL statement



    ALTER TABLE myisamtable
    ADD INDEX (column01),
    ADD INDEX (column02),
    ADD INDEX (column03),
    ...
    ADD INDEX (column10),
    ADD INDEX (column11)
    ;


    SUGGESTION #2 : Delay indexes building



    You can disable the building of non-unique indexes so that sorting and be done.



    Doing so can make a smaller BTREE for each index



    ALTER TABLE mytable DISABLE KEYS;
    ...
    ALTER TABLE mytable ENABLE KEYS;


    SUGGESTION #3 : Use a temp table



    If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.



    Here is a sample of what I just described to you



    CREATE TABLE mytable_new LIKE mytable;
    ALTER TABLE mytable_new
    ADD INDEX (column01),
    ADD INDEX (column02),
    ADD INDEX (column03),
    ...
    ADD INDEX (column10),
    ADD INDEX (column11)
    ;
    ALTER TABLE mytable_new DISABLE KEYS;
    INSERT INTO mytable_new SELECT * FROM mytable;
    ALTER TABLE mytable_new ENABLE KEYS;
    ALTER TABLE mytable RENAME mytable_old;
    ALTER TABLE mytable_new RENAME mytable;
    DROP TABLE mytable_old;


    GIVE IT A TRY !!!






    share|improve this answer















    You must be doing something odd with your DDL or you have a very, very old version of MySQL.



    MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.





    • Oct 10, 2006 : Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)


    • May 12, 2011 : Does MySQL still handle indexes in this way? (where I mention this again)


    Without any additional details from you, I can only guess. Here it goes:



    I suspect your must be just doing the following:



    ALTER TABLE mytable ADD INDEX (column01);
    ALTER TABLE mytable ADD INDEX (column02);
    ALTER TABLE mytable ADD INDEX (column03);
    ...
    ALTER TABLE mytable ADD INDEX (column10);
    ALTER TABLE mytable ADD INDEX (column11);


    I would like to suggest the following



    SUGGESTION #1 : Build all 11 indexes at the same time



    You should create all 11 indexes in a single DDL statement



    ALTER TABLE myisamtable
    ADD INDEX (column01),
    ADD INDEX (column02),
    ADD INDEX (column03),
    ...
    ADD INDEX (column10),
    ADD INDEX (column11)
    ;


    SUGGESTION #2 : Delay indexes building



    You can disable the building of non-unique indexes so that sorting and be done.



    Doing so can make a smaller BTREE for each index



    ALTER TABLE mytable DISABLE KEYS;
    ...
    ALTER TABLE mytable ENABLE KEYS;


    SUGGESTION #3 : Use a temp table



    If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.



    Here is a sample of what I just described to you



    CREATE TABLE mytable_new LIKE mytable;
    ALTER TABLE mytable_new
    ADD INDEX (column01),
    ADD INDEX (column02),
    ADD INDEX (column03),
    ...
    ADD INDEX (column10),
    ADD INDEX (column11)
    ;
    ALTER TABLE mytable_new DISABLE KEYS;
    INSERT INTO mytable_new SELECT * FROM mytable;
    ALTER TABLE mytable_new ENABLE KEYS;
    ALTER TABLE mytable RENAME mytable_old;
    ALTER TABLE mytable_new RENAME mytable;
    DROP TABLE mytable_old;


    GIVE IT A TRY !!!







    share|improve this answer














    share|improve this answer



    share|improve this answer








    edited Apr 13 '17 at 12:42









    Community

    1




    1










    answered Jun 14 '16 at 15:00









    RolandoMySQLDBARolandoMySQLDBA

    143k24226383




    143k24226383













    • If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.

      – RolandoMySQLDBA
      Jun 15 '16 at 15:41



















    • If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.

      – RolandoMySQLDBA
      Jun 15 '16 at 15:41

















    If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.

    – RolandoMySQLDBA
    Jun 15 '16 at 15:41





    If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.

    – RolandoMySQLDBA
    Jun 15 '16 at 15:41













    0














    Rolando,
    Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
    My statements were:



    CREATE INDEX idx1 ....;
    CREATE INDEX idx2 ....;
    CREATE INDEX idx3 ....;
    ...


    and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use



    ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...


    in the future.



    Thanks! Problem solved .. question answered!






    share|improve this answer




























      0














      Rolando,
      Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
      My statements were:



      CREATE INDEX idx1 ....;
      CREATE INDEX idx2 ....;
      CREATE INDEX idx3 ....;
      ...


      and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use



      ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...


      in the future.



      Thanks! Problem solved .. question answered!






      share|improve this answer


























        0












        0








        0







        Rolando,
        Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
        My statements were:



        CREATE INDEX idx1 ....;
        CREATE INDEX idx2 ....;
        CREATE INDEX idx3 ....;
        ...


        and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use



        ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...


        in the future.



        Thanks! Problem solved .. question answered!






        share|improve this answer













        Rolando,
        Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
        My statements were:



        CREATE INDEX idx1 ....;
        CREATE INDEX idx2 ....;
        CREATE INDEX idx3 ....;
        ...


        and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use



        ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...


        in the future.



        Thanks! Problem solved .. question answered!







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Jun 15 '16 at 15:38









        Wolf MetznerWolf Metzner

        61




        61






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Database Administrators Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f141192%2fmysql-create-index-slowing-down%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            الفوسفات في المغرب

            Four equal circles intersect: What is the area of the small shaded portion and its height

            جامعة ليفربول