Inserts very slow near the end of dataset
I'm trying to insert Whois Records ( domain name and relevant info) into MongoDB after parsing them from CSV files. I currently have a collection for each tLD such as .com, .org etc.
During the initial insertion of the records( such as the quick insertions of the lest populated and obscure tLDs and initial .com etc insertion ), I get decent insert speeds of about 10k-15k inserts per second.
However this slows to a crawl, part way through. Since MongoDB uses a B-tree for indexing primaries, shouldnt the performance slowdown due to database size be minimal ?
Extra Information if relevant: MongoDB 3.4, WiredTiger, Java 1.8, primary index is the domain name itself
mongodb
asked Aug 15 '17 at 19:39
Abhinav VishakAbhinav Vishak
11
bumped to the homepage by Community♦ 8 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
I'm trying to insert Whois Records ( domain name and relevant info) into MongoDB after parsing them from CSV files. I currently have a collection for each tLD such as .com, .org etc.
During the initial insertion of the records( such as the quick insertions of the lest populated and obscure tLDs and initial .com etc insertion ), I get decent insert speeds of about 10k-15k inserts per second.
However this slows to a crawl, part way through. Since MongoDB uses a B-tree for indexing primaries, shouldnt the performance slowdown due to database size be minimal ?
Extra Information if relevant: MongoDB 3.4, WiredTiger, Java 1.8, primary index is the domain name itself
mongodb
asked Aug 15 '17 at 19:39
Abhinav VishakAbhinav Vishak
11
bumped to the homepage by Community♦ 8 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
Is replication slowing it down? What is your write concern set to?
– SqlWorldWide
Aug 16 '17 at 1:05
How much RAM do you have? How large is the data set when your insertions start to slow down? A likely suspect for slowdown over time would be your data set becoming constrained by RAM or I/O. Do you have any other metrics to correlate with the period of slowdown?
– Stennie
Aug 16 '17 at 2:50
My VM is set to 8 GB currently, I don't have anything running on the server except MongoDB( though admittedly the i have to insert about 202 million records across 1000 collections, and the .com collection itself is about 133 million)
– Abhinav Vishak
Aug 16 '17 at 15:24
add a comment |
I'm trying to insert Whois Records ( domain name and relevant info) into MongoDB after parsing them from CSV files. I currently have a collection for each tLD such as .com, .org etc.
During the initial insertion of the records( such as the quick insertions of the lest populated and obscure tLDs and initial .com etc insertion ), I get decent insert speeds of about 10k-15k inserts per second.
However this slows to a crawl, part way through. Since MongoDB uses a B-tree for indexing primaries, shouldnt the performance slowdown due to database size be minimal ?
Extra Information if relevant: MongoDB 3.4, WiredTiger, Java 1.8, primary index is the domain name itself
mongodb
asked Aug 15 '17 at 19:39
Abhinav VishakAbhinav Vishak
11
I'm trying to insert Whois Records ( domain name and relevant info) into MongoDB after parsing them from CSV files. I currently have a collection for each tLD such as .com, .org etc.
During the initial insertion of the records( such as the quick insertions of the lest populated and obscure tLDs and initial .com etc insertion ), I get decent insert speeds of about 10k-15k inserts per second.
However this slows to a crawl, part way through. Since MongoDB uses a B-tree for indexing primaries, shouldnt the performance slowdown due to database size be minimal ?
Extra Information if relevant: MongoDB 3.4, WiredTiger, Java 1.8, primary index is the domain name itself
mongodb
mongodb
asked Aug 15 '17 at 19:39
Abhinav VishakAbhinav Vishak
11
asked Aug 15 '17 at 19:39
Abhinav VishakAbhinav Vishak
11
asked Aug 15 '17 at 19:39
Abhinav VishakAbhinav Vishak
11
asked Aug 15 '17 at 19:39
Abhinav VishakAbhinav Vishak
11
asked Aug 15 '17 at 19:39
Abhinav VishakAbhinav Vishak
11
11
bumped to the homepage by Community♦ 8 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 8 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
Is replication slowing it down? What is your write concern set to?
– SqlWorldWide
Aug 16 '17 at 1:05
How much RAM do you have? How large is the data set when your insertions start to slow down? A likely suspect for slowdown over time would be your data set becoming constrained by RAM or I/O. Do you have any other metrics to correlate with the period of slowdown?
– Stennie
Aug 16 '17 at 2:50
My VM is set to 8 GB currently, I don't have anything running on the server except MongoDB( though admittedly the i have to insert about 202 million records across 1000 collections, and the .com collection itself is about 133 million)
– Abhinav Vishak
Aug 16 '17 at 15:24
add a comment |
Is replication slowing it down? What is your write concern set to?
– SqlWorldWide
Aug 16 '17 at 1:05
How much RAM do you have? How large is the data set when your insertions start to slow down? A likely suspect for slowdown over time would be your data set becoming constrained by RAM or I/O. Do you have any other metrics to correlate with the period of slowdown?
– Stennie
Aug 16 '17 at 2:50
My VM is set to 8 GB currently, I don't have anything running on the server except MongoDB( though admittedly the i have to insert about 202 million records across 1000 collections, and the .com collection itself is about 133 million)
– Abhinav Vishak
Aug 16 '17 at 15:24
Is replication slowing it down? What is your write concern set to?
– SqlWorldWide
Aug 16 '17 at 1:05
Is replication slowing it down? What is your write concern set to?
– SqlWorldWide
Aug 16 '17 at 1:05
How much RAM do you have? How large is the data set when your insertions start to slow down? A likely suspect for slowdown over time would be your data set becoming constrained by RAM or I/O. Do you have any other metrics to correlate with the period of slowdown?
– Stennie
Aug 16 '17 at 2:50
How much RAM do you have? How large is the data set when your insertions start to slow down? A likely suspect for slowdown over time would be your data set becoming constrained by RAM or I/O. Do you have any other metrics to correlate with the period of slowdown?
– Stennie
Aug 16 '17 at 2:50
My VM is set to 8 GB currently, I don't have anything running on the server except MongoDB( though admittedly the i have to insert about 202 million records across 1000 collections, and the .com collection itself is about 133 million)
– Abhinav Vishak
Aug 16 '17 at 15:24
My VM is set to 8 GB currently, I don't have anything running on the server except MongoDB( though admittedly the i have to insert about 202 million records across 1000 collections, and the .com collection itself is about 133 million)
– Abhinav Vishak
Aug 16 '17 at 15:24
add a comment |
1 Answer
1
active
oldest
votes
You have run out of resources!
With 202M records, even one record would be only 64 bytes (you can check with db.collection.stats().avgObjSize), the size is 12GB and then comes index. The optimal situation is when active data, (index and data) is in the memory. This case it is NOT. So, mongod starts using disk heavily and if your disk is not high IOPS (SSD base) and low latency base, your machine starts slowing down.
Can you (for start) increase memory to (at least) 16 - 32GB?
answered Aug 17 '17 at 6:17
JJussiJJussi
3,0691314
I think i may have some additional info that is relevant. I reran the insert operations, and while monitoring with mongostat I noticed that the performance dropped when the RES/VIRT memory usage was nearing the wiredtiger default limit of 50%-1GB( around 8 GB, i had increased my ram to 16gb ). how would i go about increasing the mongod limits ?
– Abhinav Vishak
Aug 17 '17 at 20:41
also restarting mongod midway seems to "flush" the ram causing no problems, so it def looks like a RAM issue.
– Abhinav Vishak
Aug 17 '17 at 20:42
There are no real mongo limits here. Mongo will always use (eventually) all memory what is available (period). More memory is better. Until you reach the limit where all data and indexes fit in the memory and after that increasing memory don't bring any benefits.
– JJussi
Aug 18 '17 at 7:05
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "182"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f183559%2finserts-very-slow-near-the-end-of-dataset%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You have run out of resources!
With 202M records, even one record would be only 64 bytes (you can check with db.collection.stats().avgObjSize), the size is 12GB and then comes index. The optimal situation is when active data, (index and data) is in the memory. This case it is NOT. So, mongod starts using disk heavily and if your disk is not high IOPS (SSD base) and low latency base, your machine starts slowing down.
Can you (for start) increase memory to (at least) 16 - 32GB?
answered Aug 17 '17 at 6:17
JJussiJJussi
3,0691314
I think i may have some additional info that is relevant. I reran the insert operations, and while monitoring with mongostat I noticed that the performance dropped when the RES/VIRT memory usage was nearing the wiredtiger default limit of 50%-1GB( around 8 GB, i had increased my ram to 16gb ). how would i go about increasing the mongod limits ?
– Abhinav Vishak
Aug 17 '17 at 20:41
also restarting mongod midway seems to "flush" the ram causing no problems, so it def looks like a RAM issue.
– Abhinav Vishak
Aug 17 '17 at 20:42
There are no real mongo limits here. Mongo will always use (eventually) all memory what is available (period). More memory is better. Until you reach the limit where all data and indexes fit in the memory and after that increasing memory don't bring any benefits.
– JJussi
Aug 18 '17 at 7:05
add a comment |
You have run out of resources!
With 202M records, even one record would be only 64 bytes (you can check with db.collection.stats().avgObjSize), the size is 12GB and then comes index. The optimal situation is when active data, (index and data) is in the memory. This case it is NOT. So, mongod starts using disk heavily and if your disk is not high IOPS (SSD base) and low latency base, your machine starts slowing down.
Can you (for start) increase memory to (at least) 16 - 32GB?
answered Aug 17 '17 at 6:17
JJussiJJussi
3,0691314
I think i may have some additional info that is relevant. I reran the insert operations, and while monitoring with mongostat I noticed that the performance dropped when the RES/VIRT memory usage was nearing the wiredtiger default limit of 50%-1GB( around 8 GB, i had increased my ram to 16gb ). how would i go about increasing the mongod limits ?
– Abhinav Vishak
Aug 17 '17 at 20:41
also restarting mongod midway seems to "flush" the ram causing no problems, so it def looks like a RAM issue.
– Abhinav Vishak
Aug 17 '17 at 20:42
There are no real mongo limits here. Mongo will always use (eventually) all memory what is available (period). More memory is better. Until you reach the limit where all data and indexes fit in the memory and after that increasing memory don't bring any benefits.
– JJussi
Aug 18 '17 at 7:05
add a comment |
You have run out of resources!
With 202M records, even one record would be only 64 bytes (you can check with db.collection.stats().avgObjSize), the size is 12GB and then comes index. The optimal situation is when active data, (index and data) is in the memory. This case it is NOT. So, mongod starts using disk heavily and if your disk is not high IOPS (SSD base) and low latency base, your machine starts slowing down.
Can you (for start) increase memory to (at least) 16 - 32GB?
answered Aug 17 '17 at 6:17
JJussiJJussi
3,0691314
You have run out of resources!
With 202M records, even one record would be only 64 bytes (you can check with db.collection.stats().avgObjSize), the size is 12GB and then comes index. The optimal situation is when active data, (index and data) is in the memory. This case it is NOT. So, mongod starts using disk heavily and if your disk is not high IOPS (SSD base) and low latency base, your machine starts slowing down.
Can you (for start) increase memory to (at least) 16 - 32GB?
answered Aug 17 '17 at 6:17
JJussiJJussi
3,0691314
answered Aug 17 '17 at 6:17
JJussiJJussi
3,0691314
answered Aug 17 '17 at 6:17
JJussiJJussi
3,0691314
answered Aug 17 '17 at 6:17
JJussiJJussi
3,0691314
3,0691314
I think i may have some additional info that is relevant. I reran the insert operations, and while monitoring with mongostat I noticed that the performance dropped when the RES/VIRT memory usage was nearing the wiredtiger default limit of 50%-1GB( around 8 GB, i had increased my ram to 16gb ). how would i go about increasing the mongod limits ?
– Abhinav Vishak
Aug 17 '17 at 20:41
also restarting mongod midway seems to "flush" the ram causing no problems, so it def looks like a RAM issue.
– Abhinav Vishak
Aug 17 '17 at 20:42
There are no real mongo limits here. Mongo will always use (eventually) all memory what is available (period). More memory is better. Until you reach the limit where all data and indexes fit in the memory and after that increasing memory don't bring any benefits.
– JJussi
Aug 18 '17 at 7:05
add a comment |
I think i may have some additional info that is relevant. I reran the insert operations, and while monitoring with mongostat I noticed that the performance dropped when the RES/VIRT memory usage was nearing the wiredtiger default limit of 50%-1GB( around 8 GB, i had increased my ram to 16gb ). how would i go about increasing the mongod limits ?
– Abhinav Vishak
Aug 17 '17 at 20:41
also restarting mongod midway seems to "flush" the ram causing no problems, so it def looks like a RAM issue.
– Abhinav Vishak
Aug 17 '17 at 20:42
There are no real mongo limits here. Mongo will always use (eventually) all memory what is available (period). More memory is better. Until you reach the limit where all data and indexes fit in the memory and after that increasing memory don't bring any benefits.
– JJussi
Aug 18 '17 at 7:05
I think i may have some additional info that is relevant. I reran the insert operations, and while monitoring with mongostat I noticed that the performance dropped when the RES/VIRT memory usage was nearing the wiredtiger default limit of 50%-1GB( around 8 GB, i had increased my ram to 16gb ). how would i go about increasing the mongod limits ?
– Abhinav Vishak
Aug 17 '17 at 20:41
I think i may have some additional info that is relevant. I reran the insert operations, and while monitoring with mongostat I noticed that the performance dropped when the RES/VIRT memory usage was nearing the wiredtiger default limit of 50%-1GB( around 8 GB, i had increased my ram to 16gb ). how would i go about increasing the mongod limits ?
– Abhinav Vishak
Aug 17 '17 at 20:41
also restarting mongod midway seems to "flush" the ram causing no problems, so it def looks like a RAM issue.
– Abhinav Vishak
Aug 17 '17 at 20:42
also restarting mongod midway seems to "flush" the ram causing no problems, so it def looks like a RAM issue.
– Abhinav Vishak
Aug 17 '17 at 20:42
There are no real mongo limits here. Mongo will always use (eventually) all memory what is available (period). More memory is better. Until you reach the limit where all data and indexes fit in the memory and after that increasing memory don't bring any benefits.
– JJussi
Aug 18 '17 at 7:05
There are no real mongo limits here. Mongo will always use (eventually) all memory what is available (period). More memory is better. Until you reach the limit where all data and indexes fit in the memory and after that increasing memory don't bring any benefits.
– JJussi
Aug 18 '17 at 7:05
add a comment |
Thanks for contributing an answer to Database Administrators Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f183559%2finserts-very-slow-near-the-end-of-dataset%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Is replication slowing it down? What is your write concern set to?
– SqlWorldWide
Aug 16 '17 at 1:05
How much RAM do you have? How large is the data set when your insertions start to slow down? A likely suspect for slowdown over time would be your data set becoming constrained by RAM or I/O. Do you have any other metrics to correlate with the period of slowdown?
– Stennie
Aug 16 '17 at 2:50
My VM is set to 8 GB currently, I don't have anything running on the server except MongoDB( though admittedly the i have to insert about 202 million records across 1000 collections, and the .com collection itself is about 133 million)
– Abhinav Vishak
Aug 16 '17 at 15:24