Apache Kudu TServer goes down when I use CTAS (Create Table As) hence my insertion fails
I have a situation where I have a Table in Cloudera Impala (Parquet Format),
The table statistcs are:
Size: 23GB
Rows: 67M
RowSize: Approx 5KB
Columns: 308
My Cloudera is
Total 6 Nodes Cloudera Cluster (Disk : 84TB Each, Ram: 251GB Each)
Kudu Master and Tablet Server
2 Master Nodes, 5 Tablet Servers (One Node acts as a Tablet Server as well as Master)
here is my table Schema(Structure)
CREATE TABLE SRV_REQ_X
PRIMARY KEY (row_id)
PARTITION BY HASH(row_id) PARTITIONS 5
STORED AS KUDU
TBLPROPERTIES ('kudu.table_name'='IMPALA_DATABASE.KUDU TABLE NAME','kudu.master_addresses'='host1:7051,host2:7051','kudu.num_tablet_replicas' = '3')
AS
Select columns* from table*
Different Properties tested
The properties I have checked and played with are
memory_limit_hard_bytes = Checked with 0 and 1 and 250GB (Same result Tablet
Server Crashes)
maintenance_manager_num = Checked with 1 as well as 4
Records are inserted but at some point this error Comes
Kudu error(s) reported, first error: Timed out: Failed to write batch of 94 ops to tablet 842e935e768f4a419b193e1fb18e3155 after 329 attempt(s): Failed to write to server: 2d35eb2445e747bea574a5e1af6e0b2a (bda-ptcl1node02.ptcl.net.pk:7050): Write RPC to 192.168.228.2:7050 timed out after 179.996s (SENT)
I need to insert other tables which are around 102M records and I cannot understand how to tweak Kudu Properties against my Cluster.
P.S The most records went into the Kudu Table were 13M with Following Properties and then the timeout happened.
memory_limit_hard_bytes = 250GB
maintenance_manager_num = 4
block_cache_capacity_mb = 130GB
Partitions: 4
Please Help!!
bigdata cloudera impala apache-kudu
asked Nov 23 '18 at 16:01
Shahab NiazShahab Niaz
337
add a comment |
I have a situation where I have a Table in Cloudera Impala (Parquet Format),
The table statistcs are:
Size: 23GB
Rows: 67M
RowSize: Approx 5KB
Columns: 308
My Cloudera is
Total 6 Nodes Cloudera Cluster (Disk : 84TB Each, Ram: 251GB Each)
Kudu Master and Tablet Server
2 Master Nodes, 5 Tablet Servers (One Node acts as a Tablet Server as well as Master)
here is my table Schema(Structure)
CREATE TABLE SRV_REQ_X
PRIMARY KEY (row_id)
PARTITION BY HASH(row_id) PARTITIONS 5
STORED AS KUDU
TBLPROPERTIES ('kudu.table_name'='IMPALA_DATABASE.KUDU TABLE NAME','kudu.master_addresses'='host1:7051,host2:7051','kudu.num_tablet_replicas' = '3')
AS
Select columns* from table*
Different Properties tested
The properties I have checked and played with are
memory_limit_hard_bytes = Checked with 0 and 1 and 250GB (Same result Tablet
Server Crashes)
maintenance_manager_num = Checked with 1 as well as 4
Records are inserted but at some point this error Comes
Kudu error(s) reported, first error: Timed out: Failed to write batch of 94 ops to tablet 842e935e768f4a419b193e1fb18e3155 after 329 attempt(s): Failed to write to server: 2d35eb2445e747bea574a5e1af6e0b2a (bda-ptcl1node02.ptcl.net.pk:7050): Write RPC to 192.168.228.2:7050 timed out after 179.996s (SENT)
I need to insert other tables which are around 102M records and I cannot understand how to tweak Kudu Properties against my Cluster.
P.S The most records went into the Kudu Table were 13M with Following Properties and then the timeout happened.
memory_limit_hard_bytes = 250GB
maintenance_manager_num = 4
block_cache_capacity_mb = 130GB
Partitions: 4
Please Help!!
bigdata cloudera impala apache-kudu
asked Nov 23 '18 at 16:01
Shahab NiazShahab Niaz
337
Which hardware you are using?
– Abdul Majeed
Nov 23 '18 at 16:07
Oracle Big Data Appliance Starter Pack, X-6
– Shahab Niaz
Nov 23 '18 at 16:08
add a comment |
I have a situation where I have a Table in Cloudera Impala (Parquet Format),
The table statistcs are:
Size: 23GB
Rows: 67M
RowSize: Approx 5KB
Columns: 308
My Cloudera is
Total 6 Nodes Cloudera Cluster (Disk : 84TB Each, Ram: 251GB Each)
Kudu Master and Tablet Server
2 Master Nodes, 5 Tablet Servers (One Node acts as a Tablet Server as well as Master)
here is my table Schema(Structure)
CREATE TABLE SRV_REQ_X
PRIMARY KEY (row_id)
PARTITION BY HASH(row_id) PARTITIONS 5
STORED AS KUDU
TBLPROPERTIES ('kudu.table_name'='IMPALA_DATABASE.KUDU TABLE NAME','kudu.master_addresses'='host1:7051,host2:7051','kudu.num_tablet_replicas' = '3')
AS
Select columns* from table*
Different Properties tested
The properties I have checked and played with are
memory_limit_hard_bytes = Checked with 0 and 1 and 250GB (Same result Tablet
Server Crashes)
maintenance_manager_num = Checked with 1 as well as 4
Records are inserted but at some point this error Comes
Kudu error(s) reported, first error: Timed out: Failed to write batch of 94 ops to tablet 842e935e768f4a419b193e1fb18e3155 after 329 attempt(s): Failed to write to server: 2d35eb2445e747bea574a5e1af6e0b2a (bda-ptcl1node02.ptcl.net.pk:7050): Write RPC to 192.168.228.2:7050 timed out after 179.996s (SENT)
I need to insert other tables which are around 102M records and I cannot understand how to tweak Kudu Properties against my Cluster.
P.S The most records went into the Kudu Table were 13M with Following Properties and then the timeout happened.
memory_limit_hard_bytes = 250GB
maintenance_manager_num = 4
block_cache_capacity_mb = 130GB
Partitions: 4
Please Help!!
bigdata cloudera impala apache-kudu
asked Nov 23 '18 at 16:01
Shahab NiazShahab Niaz
337
I have a situation where I have a Table in Cloudera Impala (Parquet Format),
The table statistcs are:
Size: 23GB
Rows: 67M
RowSize: Approx 5KB
Columns: 308
My Cloudera is
Total 6 Nodes Cloudera Cluster (Disk : 84TB Each, Ram: 251GB Each)
Kudu Master and Tablet Server
2 Master Nodes, 5 Tablet Servers (One Node acts as a Tablet Server as well as Master)
here is my table Schema(Structure)
CREATE TABLE SRV_REQ_X
PRIMARY KEY (row_id)
PARTITION BY HASH(row_id) PARTITIONS 5
STORED AS KUDU
TBLPROPERTIES ('kudu.table_name'='IMPALA_DATABASE.KUDU TABLE NAME','kudu.master_addresses'='host1:7051,host2:7051','kudu.num_tablet_replicas' = '3')
AS
Select columns* from table*
Different Properties tested
The properties I have checked and played with are
memory_limit_hard_bytes = Checked with 0 and 1 and 250GB (Same result Tablet
Server Crashes)
maintenance_manager_num = Checked with 1 as well as 4
Records are inserted but at some point this error Comes
Kudu error(s) reported, first error: Timed out: Failed to write batch of 94 ops to tablet 842e935e768f4a419b193e1fb18e3155 after 329 attempt(s): Failed to write to server: 2d35eb2445e747bea574a5e1af6e0b2a (bda-ptcl1node02.ptcl.net.pk:7050): Write RPC to 192.168.228.2:7050 timed out after 179.996s (SENT)
I need to insert other tables which are around 102M records and I cannot understand how to tweak Kudu Properties against my Cluster.
P.S The most records went into the Kudu Table were 13M with Following Properties and then the timeout happened.
memory_limit_hard_bytes = 250GB
maintenance_manager_num = 4
block_cache_capacity_mb = 130GB
Partitions: 4
Please Help!!
bigdata cloudera impala apache-kudu
bigdata cloudera impala apache-kudu
asked Nov 23 '18 at 16:01
Shahab NiazShahab Niaz
337
asked Nov 23 '18 at 16:01
Shahab NiazShahab Niaz
337
asked Nov 23 '18 at 16:01
Shahab NiazShahab Niaz
337
asked Nov 23 '18 at 16:01
Shahab NiazShahab Niaz
337
asked Nov 23 '18 at 16:01
Shahab NiazShahab Niaz
337
337
Which hardware you are using?
– Abdul Majeed
Nov 23 '18 at 16:07
Oracle Big Data Appliance Starter Pack, X-6
– Shahab Niaz
Nov 23 '18 at 16:08
add a comment |
Which hardware you are using?
– Abdul Majeed
Nov 23 '18 at 16:07
Oracle Big Data Appliance Starter Pack, X-6
– Shahab Niaz
Nov 23 '18 at 16:08
Which hardware you are using?
– Abdul Majeed
Nov 23 '18 at 16:07
Which hardware you are using?
– Abdul Majeed
Nov 23 '18 at 16:07
Oracle Big Data Appliance Starter Pack, X-6
– Shahab Niaz
Nov 23 '18 at 16:08
Oracle Big Data Appliance Starter Pack, X-6
– Shahab Niaz
Nov 23 '18 at 16:08
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53449753%2fapache-kudu-tserver-goes-down-when-i-use-ctas-create-table-as-hence-my-inserti%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53449753%2fapache-kudu-tserver-goes-down-when-i-use-ctas-create-table-as-hence-my-inserti%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Which hardware you are using?
– Abdul Majeed
Nov 23 '18 at 16:07
Oracle Big Data Appliance Starter Pack, X-6
– Shahab Niaz
Nov 23 '18 at 16:08