distribute by sort by /cluster by clear cut idea











up vote
0
down vote

favorite












I'm not getting a clear cut idea about distribute by sort by /cluster by in hive



according to my understanding ,multiple reducers are used when we use distribute by sort by /cluster by in hive, so that sorting happens faster



but why is that the reducers are needed in sorting of columns ,sorting can be done by a map and it does not involve any aggregate function



does it have any relation with clustered by sorted by that we use when we create a table



the problem I'm facing is,



select * from order_items cluster by order_item_order_id limit 10;



for the above query ,reducer count is not changed even though i use the command,
set mapreduce.job.reduce=4



it still remains 1



you can see here ,even though your change reducer count ,it still remains 1



though there is a post related to this , the answer given there does not clear my doubts..
thanks in advance....










share|improve this question






















  • Distribute by already implies using reducers. cluster by = distribute + sort. 'mapred.reduce.tasks' is correct property.
    – leftjoin
    Nov 21 at 7:38










  • I didn't get you , can you please elaborate, "map" does the action of filtering and sorting ,so why we need reducer here.. in the given link "youtube.com/watch?v=6r2jNcisRLw" they've used mapreduce.job.reduce to increase the reducer count.can u point me where i'm going wrong..
    – Hariharan Palanichamy
    Nov 21 at 7:49












  • he used mapreduce.job.reduces, you mapreduce.job.reduce
    – leftjoin
    Nov 21 at 8:23

















up vote
0
down vote

favorite












I'm not getting a clear cut idea about distribute by sort by /cluster by in hive



according to my understanding ,multiple reducers are used when we use distribute by sort by /cluster by in hive, so that sorting happens faster



but why is that the reducers are needed in sorting of columns ,sorting can be done by a map and it does not involve any aggregate function



does it have any relation with clustered by sorted by that we use when we create a table



the problem I'm facing is,



select * from order_items cluster by order_item_order_id limit 10;



for the above query ,reducer count is not changed even though i use the command,
set mapreduce.job.reduce=4



it still remains 1



you can see here ,even though your change reducer count ,it still remains 1



though there is a post related to this , the answer given there does not clear my doubts..
thanks in advance....










share|improve this question






















  • Distribute by already implies using reducers. cluster by = distribute + sort. 'mapred.reduce.tasks' is correct property.
    – leftjoin
    Nov 21 at 7:38










  • I didn't get you , can you please elaborate, "map" does the action of filtering and sorting ,so why we need reducer here.. in the given link "youtube.com/watch?v=6r2jNcisRLw" they've used mapreduce.job.reduce to increase the reducer count.can u point me where i'm going wrong..
    – Hariharan Palanichamy
    Nov 21 at 7:49












  • he used mapreduce.job.reduces, you mapreduce.job.reduce
    – leftjoin
    Nov 21 at 8:23















up vote
0
down vote

favorite









up vote
0
down vote

favorite











I'm not getting a clear cut idea about distribute by sort by /cluster by in hive



according to my understanding ,multiple reducers are used when we use distribute by sort by /cluster by in hive, so that sorting happens faster



but why is that the reducers are needed in sorting of columns ,sorting can be done by a map and it does not involve any aggregate function



does it have any relation with clustered by sorted by that we use when we create a table



the problem I'm facing is,



select * from order_items cluster by order_item_order_id limit 10;



for the above query ,reducer count is not changed even though i use the command,
set mapreduce.job.reduce=4



it still remains 1



you can see here ,even though your change reducer count ,it still remains 1



though there is a post related to this , the answer given there does not clear my doubts..
thanks in advance....










share|improve this question













I'm not getting a clear cut idea about distribute by sort by /cluster by in hive



according to my understanding ,multiple reducers are used when we use distribute by sort by /cluster by in hive, so that sorting happens faster



but why is that the reducers are needed in sorting of columns ,sorting can be done by a map and it does not involve any aggregate function



does it have any relation with clustered by sorted by that we use when we create a table



the problem I'm facing is,



select * from order_items cluster by order_item_order_id limit 10;



for the above query ,reducer count is not changed even though i use the command,
set mapreduce.job.reduce=4



it still remains 1



you can see here ,even though your change reducer count ,it still remains 1



though there is a post related to this , the answer given there does not clear my doubts..
thanks in advance....







hive bigdata hiveql






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 21 at 7:21









Hariharan Palanichamy

34




34












  • Distribute by already implies using reducers. cluster by = distribute + sort. 'mapred.reduce.tasks' is correct property.
    – leftjoin
    Nov 21 at 7:38










  • I didn't get you , can you please elaborate, "map" does the action of filtering and sorting ,so why we need reducer here.. in the given link "youtube.com/watch?v=6r2jNcisRLw" they've used mapreduce.job.reduce to increase the reducer count.can u point me where i'm going wrong..
    – Hariharan Palanichamy
    Nov 21 at 7:49












  • he used mapreduce.job.reduces, you mapreduce.job.reduce
    – leftjoin
    Nov 21 at 8:23




















  • Distribute by already implies using reducers. cluster by = distribute + sort. 'mapred.reduce.tasks' is correct property.
    – leftjoin
    Nov 21 at 7:38










  • I didn't get you , can you please elaborate, "map" does the action of filtering and sorting ,so why we need reducer here.. in the given link "youtube.com/watch?v=6r2jNcisRLw" they've used mapreduce.job.reduce to increase the reducer count.can u point me where i'm going wrong..
    – Hariharan Palanichamy
    Nov 21 at 7:49












  • he used mapreduce.job.reduces, you mapreduce.job.reduce
    – leftjoin
    Nov 21 at 8:23


















Distribute by already implies using reducers. cluster by = distribute + sort. 'mapred.reduce.tasks' is correct property.
– leftjoin
Nov 21 at 7:38




Distribute by already implies using reducers. cluster by = distribute + sort. 'mapred.reduce.tasks' is correct property.
– leftjoin
Nov 21 at 7:38












I didn't get you , can you please elaborate, "map" does the action of filtering and sorting ,so why we need reducer here.. in the given link "youtube.com/watch?v=6r2jNcisRLw" they've used mapreduce.job.reduce to increase the reducer count.can u point me where i'm going wrong..
– Hariharan Palanichamy
Nov 21 at 7:49






I didn't get you , can you please elaborate, "map" does the action of filtering and sorting ,so why we need reducer here.. in the given link "youtube.com/watch?v=6r2jNcisRLw" they've used mapreduce.job.reduce to increase the reducer count.can u point me where i'm going wrong..
– Hariharan Palanichamy
Nov 21 at 7:49














he used mapreduce.job.reduces, you mapreduce.job.reduce
– leftjoin
Nov 21 at 8:23






he used mapreduce.job.reduces, you mapreduce.job.reduce
– leftjoin
Nov 21 at 8:23



















active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














 

draft saved


draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53407051%2fdistribute-by-sort-by-cluster-by-clear-cut-idea%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown






























active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes
















 

draft saved


draft discarded



















































 


draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53407051%2fdistribute-by-sort-by-cluster-by-clear-cut-idea%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Berounka

Sphinx de Gizeh

Different font size/position of beamer's navigation symbols template's content depending on regular/plain...