Spark: Dataframe action really slow when upgraded from 2.1.0 to 2.2.1
up vote
0
down vote
favorite
I just upgraded spark 2.1.0 to spark 2.2.1. Has anyone seen extreme slow behavior on dataframe.filter(…).collect()
?.. specifically a collect
operation with filter
before. dataframe.collect
seems to run okay. However, dataframe.filter(…).collect()
takes forever. it contains only 2 records. and its on a unit test. When I go back to spark 2.1.0, its back to normal speed
I have looked at the thread dump and could not find an obvious cause. I have made an effort to make sure all the libraries I am using are also using Spark 2.2.1. Any suggestion would be greatly appreciated.
java scala apache-spark
add a comment |
up vote
0
down vote
favorite
I just upgraded spark 2.1.0 to spark 2.2.1. Has anyone seen extreme slow behavior on dataframe.filter(…).collect()
?.. specifically a collect
operation with filter
before. dataframe.collect
seems to run okay. However, dataframe.filter(…).collect()
takes forever. it contains only 2 records. and its on a unit test. When I go back to spark 2.1.0, its back to normal speed
I have looked at the thread dump and could not find an obvious cause. I have made an effort to make sure all the libraries I am using are also using Spark 2.2.1. Any suggestion would be greatly appreciated.
java scala apache-spark
2
need more details. it was very generic question. have you checked spark ui like stages etc...?
– Ram Ghadiyaram
8 hours ago
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I just upgraded spark 2.1.0 to spark 2.2.1. Has anyone seen extreme slow behavior on dataframe.filter(…).collect()
?.. specifically a collect
operation with filter
before. dataframe.collect
seems to run okay. However, dataframe.filter(…).collect()
takes forever. it contains only 2 records. and its on a unit test. When I go back to spark 2.1.0, its back to normal speed
I have looked at the thread dump and could not find an obvious cause. I have made an effort to make sure all the libraries I am using are also using Spark 2.2.1. Any suggestion would be greatly appreciated.
java scala apache-spark
I just upgraded spark 2.1.0 to spark 2.2.1. Has anyone seen extreme slow behavior on dataframe.filter(…).collect()
?.. specifically a collect
operation with filter
before. dataframe.collect
seems to run okay. However, dataframe.filter(…).collect()
takes forever. it contains only 2 records. and its on a unit test. When I go back to spark 2.1.0, its back to normal speed
I have looked at the thread dump and could not find an obvious cause. I have made an effort to make sure all the libraries I am using are also using Spark 2.2.1. Any suggestion would be greatly appreciated.
java scala apache-spark
java scala apache-spark
asked 8 hours ago
Karan Gupta
112
112
2
need more details. it was very generic question. have you checked spark ui like stages etc...?
– Ram Ghadiyaram
8 hours ago
add a comment |
2
need more details. it was very generic question. have you checked spark ui like stages etc...?
– Ram Ghadiyaram
8 hours ago
2
2
need more details. it was very generic question. have you checked spark ui like stages etc...?
– Ram Ghadiyaram
8 hours ago
need more details. it was very generic question. have you checked spark ui like stages etc...?
– Ram Ghadiyaram
8 hours ago
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53400713%2fspark-dataframe-action-really-slow-when-upgraded-from-2-1-0-to-2-2-1%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
need more details. it was very generic question. have you checked spark ui like stages etc...?
– Ram Ghadiyaram
8 hours ago