PostgreSQL exits the shell when importing large data
I'm having this totally weird scenario: We have an archiving server based on PostgreSQL inside a Scientific Linux virtual server. I want to migrate the data (20+ GB) to a physical Scientific Linux machine, so I used
copy (select * from table) to '/path/to/file
This ran successfully, and the data is in the output file.
On the physical machine I try to import the data using
copy table from '/path/to/file'
When I try this command on a terminal emulator, it exits after a few minutes. Also, tried it on the terminal itself after login but after a few minutes it exits, too. In both cases there is no data imported.
Could it be that the server is somehow overloaded? Can we overcome this and import the entire file?
Edit: I increased the RAM to 3 GB, the crash took a bit longer to come.
Edit 2: There is a log file under /var/lib/pgsql/data/pg_log, this log file has these lines as the last few lines of it:
Context: COPY sample, line 77324116
Statement: COPY sample from STDIN
ERROR: Unexpected EOF on client connection
Context: COPY sample, line 77324116
Statement: COPY sample from STDIN
Log: Could not send data to client: Broken pipe
Statement: COPY sample from STDIN
ERROR: Unexpected EOF on client connection
Any help is really appreciated!
postgresql copy
asked Sep 22 '16 at 7:01
3bdalla3bdalla
13426
bumped to the homepage by Community♦ 11 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
|
show 1 more comment
I'm having this totally weird scenario: We have an archiving server based on PostgreSQL inside a Scientific Linux virtual server. I want to migrate the data (20+ GB) to a physical Scientific Linux machine, so I used
copy (select * from table) to '/path/to/file
This ran successfully, and the data is in the output file.
On the physical machine I try to import the data using
copy table from '/path/to/file'
When I try this command on a terminal emulator, it exits after a few minutes. Also, tried it on the terminal itself after login but after a few minutes it exits, too. In both cases there is no data imported.
Could it be that the server is somehow overloaded? Can we overcome this and import the entire file?
Edit: I increased the RAM to 3 GB, the crash took a bit longer to come.
Edit 2: There is a log file under /var/lib/pgsql/data/pg_log, this log file has these lines as the last few lines of it:
Context: COPY sample, line 77324116
Statement: COPY sample from STDIN
ERROR: Unexpected EOF on client connection
Context: COPY sample, line 77324116
Statement: COPY sample from STDIN
Log: Could not send data to client: Broken pipe
Statement: COPY sample from STDIN
ERROR: Unexpected EOF on client connection
Any help is really appreciated!
postgresql copy
asked Sep 22 '16 at 7:01
3bdalla3bdalla
13426
bumped to the homepage by Community♦ 11 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
2
Usescreen. You might be a victim of some network problems. Also, please note thatpsqlis a client to PostgreSQL, not just another abbreviation for it.
– dezso
Sep 22 '16 at 8:13
I'm importing the data to the server on the same machine, but the data file exists on a network drive.
– 3bdalla
Sep 22 '16 at 8:45
3
What does the server log file say? It will usually have more detailed information than the client log file does. You should also check the system's log files, to see if you are getting out-of-memory problems.
– jjanes
Sep 22 '16 at 19:36
Check my update.
– 3bdalla
Sep 25 '16 at 10:24
Are you operating over a SSH connection?
– pietrop
Sep 28 '16 at 15:35
|
show 1 more comment
I'm having this totally weird scenario: We have an archiving server based on PostgreSQL inside a Scientific Linux virtual server. I want to migrate the data (20+ GB) to a physical Scientific Linux machine, so I used
copy (select * from table) to '/path/to/file
This ran successfully, and the data is in the output file.
On the physical machine I try to import the data using
copy table from '/path/to/file'
When I try this command on a terminal emulator, it exits after a few minutes. Also, tried it on the terminal itself after login but after a few minutes it exits, too. In both cases there is no data imported.
Could it be that the server is somehow overloaded? Can we overcome this and import the entire file?
Edit: I increased the RAM to 3 GB, the crash took a bit longer to come.
Edit 2: There is a log file under /var/lib/pgsql/data/pg_log, this log file has these lines as the last few lines of it:
Context: COPY sample, line 77324116
Statement: COPY sample from STDIN
ERROR: Unexpected EOF on client connection
Context: COPY sample, line 77324116
Statement: COPY sample from STDIN
Log: Could not send data to client: Broken pipe
Statement: COPY sample from STDIN
ERROR: Unexpected EOF on client connection
Any help is really appreciated!
postgresql copy
asked Sep 22 '16 at 7:01
3bdalla3bdalla
13426
I'm having this totally weird scenario: We have an archiving server based on PostgreSQL inside a Scientific Linux virtual server. I want to migrate the data (20+ GB) to a physical Scientific Linux machine, so I used
copy (select * from table) to '/path/to/file
This ran successfully, and the data is in the output file.
On the physical machine I try to import the data using
copy table from '/path/to/file'
When I try this command on a terminal emulator, it exits after a few minutes. Also, tried it on the terminal itself after login but after a few minutes it exits, too. In both cases there is no data imported.
Could it be that the server is somehow overloaded? Can we overcome this and import the entire file?
Edit: I increased the RAM to 3 GB, the crash took a bit longer to come.
Edit 2: There is a log file under /var/lib/pgsql/data/pg_log, this log file has these lines as the last few lines of it:
Context: COPY sample, line 77324116
Statement: COPY sample from STDIN
ERROR: Unexpected EOF on client connection
Context: COPY sample, line 77324116
Statement: COPY sample from STDIN
Log: Could not send data to client: Broken pipe
Statement: COPY sample from STDIN
ERROR: Unexpected EOF on client connection
Any help is really appreciated!
postgresql copy
postgresql copy
asked Sep 22 '16 at 7:01
3bdalla3bdalla
13426
asked Sep 22 '16 at 7:01
3bdalla3bdalla
13426
edited Sep 25 '16 at 10:24
3bdalla
asked Sep 22 '16 at 7:01
3bdalla3bdalla
13426
asked Sep 22 '16 at 7:01
3bdalla3bdalla
13426
asked Sep 22 '16 at 7:01
3bdalla3bdalla
13426
13426
bumped to the homepage by Community♦ 11 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 11 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
2
Usescreen. You might be a victim of some network problems. Also, please note thatpsqlis a client to PostgreSQL, not just another abbreviation for it.
– dezso
Sep 22 '16 at 8:13
I'm importing the data to the server on the same machine, but the data file exists on a network drive.
– 3bdalla
Sep 22 '16 at 8:45
3
What does the server log file say? It will usually have more detailed information than the client log file does. You should also check the system's log files, to see if you are getting out-of-memory problems.
– jjanes
Sep 22 '16 at 19:36
Check my update.
– 3bdalla
Sep 25 '16 at 10:24
Are you operating over a SSH connection?
– pietrop
Sep 28 '16 at 15:35
|
show 1 more comment
2
Usescreen. You might be a victim of some network problems. Also, please note thatpsqlis a client to PostgreSQL, not just another abbreviation for it.
– dezso
Sep 22 '16 at 8:13
I'm importing the data to the server on the same machine, but the data file exists on a network drive.
– 3bdalla
Sep 22 '16 at 8:45
3
What does the server log file say? It will usually have more detailed information than the client log file does. You should also check the system's log files, to see if you are getting out-of-memory problems.
– jjanes
Sep 22 '16 at 19:36
Check my update.
– 3bdalla
Sep 25 '16 at 10:24
Are you operating over a SSH connection?
– pietrop
Sep 28 '16 at 15:35
2
2
Use
screen. You might be a victim of some network problems. Also, please note that psql is a client to PostgreSQL, not just another abbreviation for it.– dezso
Sep 22 '16 at 8:13
Use
screen. You might be a victim of some network problems. Also, please note that psql is a client to PostgreSQL, not just another abbreviation for it.– dezso
Sep 22 '16 at 8:13
I'm importing the data to the server on the same machine, but the data file exists on a network drive.
– 3bdalla
Sep 22 '16 at 8:45
I'm importing the data to the server on the same machine, but the data file exists on a network drive.
– 3bdalla
Sep 22 '16 at 8:45
3
3
What does the server log file say? It will usually have more detailed information than the client log file does. You should also check the system's log files, to see if you are getting out-of-memory problems.
– jjanes
Sep 22 '16 at 19:36
What does the server log file say? It will usually have more detailed information than the client log file does. You should also check the system's log files, to see if you are getting out-of-memory problems.
– jjanes
Sep 22 '16 at 19:36
Check my update.
– 3bdalla
Sep 25 '16 at 10:24
Check my update.
– 3bdalla
Sep 25 '16 at 10:24
Are you operating over a SSH connection?
– pietrop
Sep 28 '16 at 15:35
Are you operating over a SSH connection?
– pietrop
Sep 28 '16 at 15:35
|
show 1 more comment
1 Answer
1
active
oldest
votes
Is your physical machine using autologout?
$ echo $TMOUT
When there exists a value above zero, you should unset that environment variable.
$ unset TMOUT
http://www.cyberciti.biz/faq/unset-tmout-under-unix-appleosx-bash-ksh-tcsh-shell/
answered Sep 22 '16 at 23:11
maletinmaletin
1279
This env variable is not set on my machine. Check my update on the question.
– 3bdalla
Sep 25 '16 at 10:24
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "182"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f150369%2fpostgresql-exits-the-shell-when-importing-large-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Is your physical machine using autologout?
$ echo $TMOUT
When there exists a value above zero, you should unset that environment variable.
$ unset TMOUT
http://www.cyberciti.biz/faq/unset-tmout-under-unix-appleosx-bash-ksh-tcsh-shell/
answered Sep 22 '16 at 23:11
maletinmaletin
1279
This env variable is not set on my machine. Check my update on the question.
– 3bdalla
Sep 25 '16 at 10:24
add a comment |
Is your physical machine using autologout?
$ echo $TMOUT
When there exists a value above zero, you should unset that environment variable.
$ unset TMOUT
http://www.cyberciti.biz/faq/unset-tmout-under-unix-appleosx-bash-ksh-tcsh-shell/
answered Sep 22 '16 at 23:11
maletinmaletin
1279
This env variable is not set on my machine. Check my update on the question.
– 3bdalla
Sep 25 '16 at 10:24
add a comment |
Is your physical machine using autologout?
$ echo $TMOUT
When there exists a value above zero, you should unset that environment variable.
$ unset TMOUT
http://www.cyberciti.biz/faq/unset-tmout-under-unix-appleosx-bash-ksh-tcsh-shell/
answered Sep 22 '16 at 23:11
maletinmaletin
1279
Is your physical machine using autologout?
$ echo $TMOUT
When there exists a value above zero, you should unset that environment variable.
$ unset TMOUT
http://www.cyberciti.biz/faq/unset-tmout-under-unix-appleosx-bash-ksh-tcsh-shell/
answered Sep 22 '16 at 23:11
maletinmaletin
1279
answered Sep 22 '16 at 23:11
maletinmaletin
1279
answered Sep 22 '16 at 23:11
maletinmaletin
1279
answered Sep 22 '16 at 23:11
maletinmaletin
1279
1279
This env variable is not set on my machine. Check my update on the question.
– 3bdalla
Sep 25 '16 at 10:24
add a comment |
This env variable is not set on my machine. Check my update on the question.
– 3bdalla
Sep 25 '16 at 10:24
This env variable is not set on my machine. Check my update on the question.
– 3bdalla
Sep 25 '16 at 10:24
This env variable is not set on my machine. Check my update on the question.
– 3bdalla
Sep 25 '16 at 10:24
add a comment |
Thanks for contributing an answer to Database Administrators Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f150369%2fpostgresql-exits-the-shell-when-importing-large-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
Use
screen. You might be a victim of some network problems. Also, please note thatpsqlis a client to PostgreSQL, not just another abbreviation for it.– dezso
Sep 22 '16 at 8:13
I'm importing the data to the server on the same machine, but the data file exists on a network drive.
– 3bdalla
Sep 22 '16 at 8:45
3
What does the server log file say? It will usually have more detailed information than the client log file does. You should also check the system's log files, to see if you are getting out-of-memory problems.
– jjanes
Sep 22 '16 at 19:36
Check my update.
– 3bdalla
Sep 25 '16 at 10:24
Are you operating over a SSH connection?
– pietrop
Sep 28 '16 at 15:35