PySpark: how to resolve path of a resource file present inside the dependency zip file
I have a mapPartitions on an RDD and within each partition, a resource file has to be opened. This module that contains the method invoked by mapPartitions and the resource file is passed on to each executor using the --py-files argument as a zip file.
To make it clear:
rdd = rdd.mapPartitions(work_doing_method)
def work_doing_method(rows):
for row in rows:
resource_file_path = os.path.join(os.path.dirname(__file__), "resource.json")
with open(resource_file_path) as f:
resource = json.loads(f.read())
...
When I do this after passing the zip file which includes all of this using the --py-file parameter to the spark-submit command,
I get IOError: [Errno 20] Not a directory:/full/path/to/the/file/within/zip/file
I do not understand how Spark uses the zip file to read the dependencies. The os.path.dirname utility returns the full path including the zip file, for eg. /spark/dir/my_dependency_file.zip/path/to/the/resource/file. I believe this should be the problem. I tried many combinations to resolve the path of the file. Any help is appreciated.
Thanks!
python apache-spark pyspark
asked Nov 13 '18 at 12:45
voidvoid
1,03221737
add a comment |
I have a mapPartitions on an RDD and within each partition, a resource file has to be opened. This module that contains the method invoked by mapPartitions and the resource file is passed on to each executor using the --py-files argument as a zip file.
To make it clear:
rdd = rdd.mapPartitions(work_doing_method)
def work_doing_method(rows):
for row in rows:
resource_file_path = os.path.join(os.path.dirname(__file__), "resource.json")
with open(resource_file_path) as f:
resource = json.loads(f.read())
...
When I do this after passing the zip file which includes all of this using the --py-file parameter to the spark-submit command,
I get IOError: [Errno 20] Not a directory:/full/path/to/the/file/within/zip/file
I do not understand how Spark uses the zip file to read the dependencies. The os.path.dirname utility returns the full path including the zip file, for eg. /spark/dir/my_dependency_file.zip/path/to/the/resource/file. I believe this should be the problem. I tried many combinations to resolve the path of the file. Any help is appreciated.
Thanks!
python apache-spark pyspark
asked Nov 13 '18 at 12:45
voidvoid
1,03221737
add a comment |
I have a mapPartitions on an RDD and within each partition, a resource file has to be opened. This module that contains the method invoked by mapPartitions and the resource file is passed on to each executor using the --py-files argument as a zip file.
To make it clear:
rdd = rdd.mapPartitions(work_doing_method)
def work_doing_method(rows):
for row in rows:
resource_file_path = os.path.join(os.path.dirname(__file__), "resource.json")
with open(resource_file_path) as f:
resource = json.loads(f.read())
...
When I do this after passing the zip file which includes all of this using the --py-file parameter to the spark-submit command,
I get IOError: [Errno 20] Not a directory:/full/path/to/the/file/within/zip/file
I do not understand how Spark uses the zip file to read the dependencies. The os.path.dirname utility returns the full path including the zip file, for eg. /spark/dir/my_dependency_file.zip/path/to/the/resource/file. I believe this should be the problem. I tried many combinations to resolve the path of the file. Any help is appreciated.
Thanks!
python apache-spark pyspark
asked Nov 13 '18 at 12:45
voidvoid
1,03221737
I have a mapPartitions on an RDD and within each partition, a resource file has to be opened. This module that contains the method invoked by mapPartitions and the resource file is passed on to each executor using the --py-files argument as a zip file.
To make it clear:
rdd = rdd.mapPartitions(work_doing_method)
def work_doing_method(rows):
for row in rows:
resource_file_path = os.path.join(os.path.dirname(__file__), "resource.json")
with open(resource_file_path) as f:
resource = json.loads(f.read())
...
When I do this after passing the zip file which includes all of this using the --py-file parameter to the spark-submit command,
I get IOError: [Errno 20] Not a directory:/full/path/to/the/file/within/zip/file
I do not understand how Spark uses the zip file to read the dependencies. The os.path.dirname utility returns the full path including the zip file, for eg. /spark/dir/my_dependency_file.zip/path/to/the/resource/file. I believe this should be the problem. I tried many combinations to resolve the path of the file. Any help is appreciated.
Thanks!
python apache-spark pyspark
python apache-spark pyspark
asked Nov 13 '18 at 12:45
voidvoid
1,03221737
asked Nov 13 '18 at 12:45
voidvoid
1,03221737
edited Nov 13 '18 at 14:22
void
asked Nov 13 '18 at 12:45
voidvoid
1,03221737
asked Nov 13 '18 at 12:45
voidvoid
1,03221737
asked Nov 13 '18 at 12:45
voidvoid
1,03221737
1,03221737
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
I think when you add a file to a Spark job, it will be copied to the working directory of each executor. I've used the SparkFiles API to get absolute paths to files on the executors.
You can also use the --archives flag to pass in arbitrary data archives such as zipfiles. What's the difference between --archives, --files, py-files in pyspark job arguments
answered Nov 13 '18 at 12:58
Graham HealyGraham Healy
515
The spark files API is a good solution but it will break the abstraction of the dependency module that we have (the absolute path has to be passed on to the dependency module from the spark job client). I am currently exploring packaging as an egg file or a wheel file (which could bunde data files as well?), so far no luck though.
– void
Nov 14 '18 at 8:45
add a comment |
We get the path to a resource file within an egg/zip file (inside the executor working dir) when we look for the absolute path. I ended up using the zipfile module in Python and actually open it like here.
answered Nov 23 '18 at 10:43
voidvoid
1,03221737
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53281314%2fpyspark-how-to-resolve-path-of-a-resource-file-present-inside-the-dependency-zi%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
I think when you add a file to a Spark job, it will be copied to the working directory of each executor. I've used the SparkFiles API to get absolute paths to files on the executors.
You can also use the --archives flag to pass in arbitrary data archives such as zipfiles. What's the difference between --archives, --files, py-files in pyspark job arguments
answered Nov 13 '18 at 12:58
Graham HealyGraham Healy
515
The spark files API is a good solution but it will break the abstraction of the dependency module that we have (the absolute path has to be passed on to the dependency module from the spark job client). I am currently exploring packaging as an egg file or a wheel file (which could bunde data files as well?), so far no luck though.
– void
Nov 14 '18 at 8:45
add a comment |
I think when you add a file to a Spark job, it will be copied to the working directory of each executor. I've used the SparkFiles API to get absolute paths to files on the executors.
You can also use the --archives flag to pass in arbitrary data archives such as zipfiles. What's the difference between --archives, --files, py-files in pyspark job arguments
answered Nov 13 '18 at 12:58
Graham HealyGraham Healy
515
The spark files API is a good solution but it will break the abstraction of the dependency module that we have (the absolute path has to be passed on to the dependency module from the spark job client). I am currently exploring packaging as an egg file or a wheel file (which could bunde data files as well?), so far no luck though.
– void
Nov 14 '18 at 8:45
add a comment |
I think when you add a file to a Spark job, it will be copied to the working directory of each executor. I've used the SparkFiles API to get absolute paths to files on the executors.
You can also use the --archives flag to pass in arbitrary data archives such as zipfiles. What's the difference between --archives, --files, py-files in pyspark job arguments
answered Nov 13 '18 at 12:58
Graham HealyGraham Healy
515
I think when you add a file to a Spark job, it will be copied to the working directory of each executor. I've used the SparkFiles API to get absolute paths to files on the executors.
You can also use the --archives flag to pass in arbitrary data archives such as zipfiles. What's the difference between --archives, --files, py-files in pyspark job arguments
answered Nov 13 '18 at 12:58
Graham HealyGraham Healy
515
answered Nov 13 '18 at 12:58
Graham HealyGraham Healy
515
answered Nov 13 '18 at 12:58
Graham HealyGraham Healy
515
answered Nov 13 '18 at 12:58
Graham HealyGraham Healy
515
515
The spark files API is a good solution but it will break the abstraction of the dependency module that we have (the absolute path has to be passed on to the dependency module from the spark job client). I am currently exploring packaging as an egg file or a wheel file (which could bunde data files as well?), so far no luck though.
– void
Nov 14 '18 at 8:45
add a comment |
The spark files API is a good solution but it will break the abstraction of the dependency module that we have (the absolute path has to be passed on to the dependency module from the spark job client). I am currently exploring packaging as an egg file or a wheel file (which could bunde data files as well?), so far no luck though.
– void
Nov 14 '18 at 8:45
The spark files API is a good solution but it will break the abstraction of the dependency module that we have (the absolute path has to be passed on to the dependency module from the spark job client). I am currently exploring packaging as an egg file or a wheel file (which could bunde data files as well?), so far no luck though.
– void
Nov 14 '18 at 8:45
The spark files API is a good solution but it will break the abstraction of the dependency module that we have (the absolute path has to be passed on to the dependency module from the spark job client). I am currently exploring packaging as an egg file or a wheel file (which could bunde data files as well?), so far no luck though.
– void
Nov 14 '18 at 8:45
add a comment |
We get the path to a resource file within an egg/zip file (inside the executor working dir) when we look for the absolute path. I ended up using the zipfile module in Python and actually open it like here.
answered Nov 23 '18 at 10:43
voidvoid
1,03221737
add a comment |
We get the path to a resource file within an egg/zip file (inside the executor working dir) when we look for the absolute path. I ended up using the zipfile module in Python and actually open it like here.
answered Nov 23 '18 at 10:43
voidvoid
1,03221737
add a comment |
We get the path to a resource file within an egg/zip file (inside the executor working dir) when we look for the absolute path. I ended up using the zipfile module in Python and actually open it like here.
answered Nov 23 '18 at 10:43
voidvoid
1,03221737
We get the path to a resource file within an egg/zip file (inside the executor working dir) when we look for the absolute path. I ended up using the zipfile module in Python and actually open it like here.
answered Nov 23 '18 at 10:43
voidvoid
1,03221737
edited Nov 23 '18 at 11:47
answered Nov 23 '18 at 10:43
voidvoid
1,03221737
answered Nov 23 '18 at 10:43
voidvoid
1,03221737
answered Nov 23 '18 at 10:43
voidvoid
1,03221737
1,03221737
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53281314%2fpyspark-how-to-resolve-path-of-a-resource-file-present-inside-the-dependency-zi%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
