Spark RDD Windowing using pyspark
There is a Spark RDD, called rdd1. It has(key, value) pair and I have a list, whose elements are a tuple(key1,key2).
I want to get a rdd2, with rows `((key1,key2), (value of key1 in rdd1, value of key2 in rdd1)).
Can somebody help me?
rdd1:
key1, value1,
key2, value2,
key3, value3
array: [(key1,key2),(key2,key3)]
Result:
(key1,key2),value1,value2
(key2,key3),value2,value3
I have tried
spark.parallize(array).map(lambda x:)
apache-spark join pyspark rdd
edited Nov 23 at 11:32
thebluephantom
2,3652925
asked Nov 22 at 13:35
user9465775
6
add a comment |
There is a Spark RDD, called rdd1. It has(key, value) pair and I have a list, whose elements are a tuple(key1,key2).
I want to get a rdd2, with rows `((key1,key2), (value of key1 in rdd1, value of key2 in rdd1)).
Can somebody help me?
rdd1:
key1, value1,
key2, value2,
key3, value3
array: [(key1,key2),(key2,key3)]
Result:
(key1,key2),value1,value2
(key2,key3),value2,value3
I have tried
spark.parallize(array).map(lambda x:)
apache-spark join pyspark rdd
edited Nov 23 at 11:32
thebluephantom
2,3652925
asked Nov 22 at 13:35
user9465775
6
add a comment |
There is a Spark RDD, called rdd1. It has(key, value) pair and I have a list, whose elements are a tuple(key1,key2).
I want to get a rdd2, with rows `((key1,key2), (value of key1 in rdd1, value of key2 in rdd1)).
Can somebody help me?
rdd1:
key1, value1,
key2, value2,
key3, value3
array: [(key1,key2),(key2,key3)]
Result:
(key1,key2),value1,value2
(key2,key3),value2,value3
I have tried
spark.parallize(array).map(lambda x:)
apache-spark join pyspark rdd
edited Nov 23 at 11:32
thebluephantom
2,3652925
asked Nov 22 at 13:35
user9465775
6
There is a Spark RDD, called rdd1. It has(key, value) pair and I have a list, whose elements are a tuple(key1,key2).
I want to get a rdd2, with rows `((key1,key2), (value of key1 in rdd1, value of key2 in rdd1)).
Can somebody help me?
rdd1:
key1, value1,
key2, value2,
key3, value3
array: [(key1,key2),(key2,key3)]
Result:
(key1,key2),value1,value2
(key2,key3),value2,value3
I have tried
spark.parallize(array).map(lambda x:)
apache-spark join pyspark rdd
apache-spark join pyspark rdd
edited Nov 23 at 11:32
thebluephantom
2,3652925
asked Nov 22 at 13:35
user9465775
6
edited Nov 23 at 11:32
thebluephantom
2,3652925
asked Nov 22 at 13:35
user9465775
6
edited Nov 23 at 11:32
thebluephantom
2,3652925
edited Nov 23 at 11:32
thebluephantom
2,3652925
edited Nov 23 at 11:32
thebluephantom
2,3652925
2,3652925
asked Nov 22 at 13:35
user9465775
6
asked Nov 22 at 13:35
user9465775
6
asked Nov 22 at 13:35
user9465775
6
6
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
sliding with SCALA vs mllib sliding - two implementations, a bit fiddly but here it is:
import org.apache.spark.mllib.rdd.RDDFunctions._
val rdd1 = sc.parallelize(Seq(
( "key1", "value1"),
( "key2", "value2"),
( "key3", "value3"),
( "key4", "value4"),
( "key5", "value5")
))
val rdd2 = rdd1.sliding(2)
val rdd3 = rdd2.map(x => (x(0), x(1)))
val rdd4 = rdd3.map(x => ((x._1._1, x._2._1),x._1._2, x._2._2))
rdd4.collect
also, the following and this is actually better of course... :
val rdd5 = rdd2.map{case Array(x,y) => ((x._1, y._1), x._2, y._2)}
rdd5.collect
returns in both cases:
res70: Array[((String, String), String, String)] = Array(((key1,key2),value1,value2), ((key2,key3),value2,value3), ((key3,key4),value3,value4), ((key4,key5),value4,value5))
which I believe meets your needs, but not in pyspark.
On Stack Overflow you can find statements that pyspark does not have an equivalent for RDDs unless you "roll your own". You could look at this How to transform data with sliding window over time series data in Pyspark. However, I would advise Data Frames with the use of pyspark.sql.functions.lead() and pyspark.sql.functions.lag(). Somewhat easier.
answered Nov 22 at 21:55
thebluephantom
2,3652925
You will need to convert to pyspark.
– thebluephantom
Nov 22 at 22:52
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53432190%2fspark-rdd-windowing-using-pyspark%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
sliding with SCALA vs mllib sliding - two implementations, a bit fiddly but here it is:
import org.apache.spark.mllib.rdd.RDDFunctions._
val rdd1 = sc.parallelize(Seq(
( "key1", "value1"),
( "key2", "value2"),
( "key3", "value3"),
( "key4", "value4"),
( "key5", "value5")
))
val rdd2 = rdd1.sliding(2)
val rdd3 = rdd2.map(x => (x(0), x(1)))
val rdd4 = rdd3.map(x => ((x._1._1, x._2._1),x._1._2, x._2._2))
rdd4.collect
also, the following and this is actually better of course... :
val rdd5 = rdd2.map{case Array(x,y) => ((x._1, y._1), x._2, y._2)}
rdd5.collect
returns in both cases:
res70: Array[((String, String), String, String)] = Array(((key1,key2),value1,value2), ((key2,key3),value2,value3), ((key3,key4),value3,value4), ((key4,key5),value4,value5))
which I believe meets your needs, but not in pyspark.
On Stack Overflow you can find statements that pyspark does not have an equivalent for RDDs unless you "roll your own". You could look at this How to transform data with sliding window over time series data in Pyspark. However, I would advise Data Frames with the use of pyspark.sql.functions.lead() and pyspark.sql.functions.lag(). Somewhat easier.
answered Nov 22 at 21:55
thebluephantom
2,3652925
You will need to convert to pyspark.
– thebluephantom
Nov 22 at 22:52
add a comment |
sliding with SCALA vs mllib sliding - two implementations, a bit fiddly but here it is:
import org.apache.spark.mllib.rdd.RDDFunctions._
val rdd1 = sc.parallelize(Seq(
( "key1", "value1"),
( "key2", "value2"),
( "key3", "value3"),
( "key4", "value4"),
( "key5", "value5")
))
val rdd2 = rdd1.sliding(2)
val rdd3 = rdd2.map(x => (x(0), x(1)))
val rdd4 = rdd3.map(x => ((x._1._1, x._2._1),x._1._2, x._2._2))
rdd4.collect
also, the following and this is actually better of course... :
val rdd5 = rdd2.map{case Array(x,y) => ((x._1, y._1), x._2, y._2)}
rdd5.collect
returns in both cases:
res70: Array[((String, String), String, String)] = Array(((key1,key2),value1,value2), ((key2,key3),value2,value3), ((key3,key4),value3,value4), ((key4,key5),value4,value5))
which I believe meets your needs, but not in pyspark.
On Stack Overflow you can find statements that pyspark does not have an equivalent for RDDs unless you "roll your own". You could look at this How to transform data with sliding window over time series data in Pyspark. However, I would advise Data Frames with the use of pyspark.sql.functions.lead() and pyspark.sql.functions.lag(). Somewhat easier.
answered Nov 22 at 21:55
thebluephantom
2,3652925
You will need to convert to pyspark.
– thebluephantom
Nov 22 at 22:52
add a comment |
sliding with SCALA vs mllib sliding - two implementations, a bit fiddly but here it is:
import org.apache.spark.mllib.rdd.RDDFunctions._
val rdd1 = sc.parallelize(Seq(
( "key1", "value1"),
( "key2", "value2"),
( "key3", "value3"),
( "key4", "value4"),
( "key5", "value5")
))
val rdd2 = rdd1.sliding(2)
val rdd3 = rdd2.map(x => (x(0), x(1)))
val rdd4 = rdd3.map(x => ((x._1._1, x._2._1),x._1._2, x._2._2))
rdd4.collect
also, the following and this is actually better of course... :
val rdd5 = rdd2.map{case Array(x,y) => ((x._1, y._1), x._2, y._2)}
rdd5.collect
returns in both cases:
res70: Array[((String, String), String, String)] = Array(((key1,key2),value1,value2), ((key2,key3),value2,value3), ((key3,key4),value3,value4), ((key4,key5),value4,value5))
which I believe meets your needs, but not in pyspark.
On Stack Overflow you can find statements that pyspark does not have an equivalent for RDDs unless you "roll your own". You could look at this How to transform data with sliding window over time series data in Pyspark. However, I would advise Data Frames with the use of pyspark.sql.functions.lead() and pyspark.sql.functions.lag(). Somewhat easier.
answered Nov 22 at 21:55
thebluephantom
2,3652925
sliding with SCALA vs mllib sliding - two implementations, a bit fiddly but here it is:
import org.apache.spark.mllib.rdd.RDDFunctions._
val rdd1 = sc.parallelize(Seq(
( "key1", "value1"),
( "key2", "value2"),
( "key3", "value3"),
( "key4", "value4"),
( "key5", "value5")
))
val rdd2 = rdd1.sliding(2)
val rdd3 = rdd2.map(x => (x(0), x(1)))
val rdd4 = rdd3.map(x => ((x._1._1, x._2._1),x._1._2, x._2._2))
rdd4.collect
also, the following and this is actually better of course... :
val rdd5 = rdd2.map{case Array(x,y) => ((x._1, y._1), x._2, y._2)}
rdd5.collect
returns in both cases:
res70: Array[((String, String), String, String)] = Array(((key1,key2),value1,value2), ((key2,key3),value2,value3), ((key3,key4),value3,value4), ((key4,key5),value4,value5))
which I believe meets your needs, but not in pyspark.
On Stack Overflow you can find statements that pyspark does not have an equivalent for RDDs unless you "roll your own". You could look at this How to transform data with sliding window over time series data in Pyspark. However, I would advise Data Frames with the use of pyspark.sql.functions.lead() and pyspark.sql.functions.lag(). Somewhat easier.
answered Nov 22 at 21:55
thebluephantom
2,3652925
edited Nov 23 at 19:32
answered Nov 22 at 21:55
thebluephantom
2,3652925
answered Nov 22 at 21:55
thebluephantom
2,3652925
answered Nov 22 at 21:55
thebluephantom
2,3652925
2,3652925
You will need to convert to pyspark.
– thebluephantom
Nov 22 at 22:52
add a comment |
You will need to convert to pyspark.
– thebluephantom
Nov 22 at 22:52
You will need to convert to pyspark.
– thebluephantom
Nov 22 at 22:52
You will need to convert to pyspark.
– thebluephantom
Nov 22 at 22:52
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53432190%2fspark-rdd-windowing-using-pyspark%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
