GCP DataflowRunner ImportErrors
Code is working when using option DirectRunner. But getting import errors when switching it to DataflowRunner. lxml module is not found is the reason. When trying to use setuptools code along with the main code, its still not working ( --setup_file setup.py).
setuptools.setup(
name='lxml',
version='4.2.5',
install_requires=,
packages= setuptools.find_packages(),
)
Error: ImportError: No module named lxml [while running 'Run Query']
Any help/suggestions to overcome this error? Thanks.
python
google-cloud-platform apache-beam edited Nov 30 '18 at 3:35
Ramesh Dharan
32016
asked Nov 23 '18 at 7:02
pavanpavan
1
add a comment |
Code is working when using option DirectRunner. But getting import errors when switching it to DataflowRunner. lxml module is not found is the reason. When trying to use setuptools code along with the main code, its still not working ( --setup_file setup.py).
setuptools.setup(
name='lxml',
version='4.2.5',
install_requires=,
packages= setuptools.find_packages(),
)
Error: ImportError: No module named lxml [while running 'Run Query']
Any help/suggestions to overcome this error? Thanks.
python
google-cloud-platform apache-beam edited Nov 30 '18 at 3:35
Ramesh Dharan
32016
asked Nov 23 '18 at 7:02
pavanpavan
1
add a comment |
Code is working when using option DirectRunner. But getting import errors when switching it to DataflowRunner. lxml module is not found is the reason. When trying to use setuptools code along with the main code, its still not working ( --setup_file setup.py).
setuptools.setup(
name='lxml',
version='4.2.5',
install_requires=,
packages= setuptools.find_packages(),
)
Error: ImportError: No module named lxml [while running 'Run Query']
Any help/suggestions to overcome this error? Thanks.
python
google-cloud-platform apache-beam edited Nov 30 '18 at 3:35
Ramesh Dharan
32016
asked Nov 23 '18 at 7:02
pavanpavan
1
Code is working when using option DirectRunner. But getting import errors when switching it to DataflowRunner. lxml module is not found is the reason. When trying to use setuptools code along with the main code, its still not working ( --setup_file setup.py).
setuptools.setup(
name='lxml',
version='4.2.5',
install_requires=,
packages= setuptools.find_packages(),
)
Error: ImportError: No module named lxml [while running 'Run Query']
Any help/suggestions to overcome this error? Thanks.
python
google-cloud-platform apache-beam python
google-cloud-platform apache-beam edited Nov 30 '18 at 3:35
Ramesh Dharan
32016
asked Nov 23 '18 at 7:02
pavanpavan
1
edited Nov 30 '18 at 3:35
Ramesh Dharan
32016
asked Nov 23 '18 at 7:02
pavanpavan
1
edited Nov 30 '18 at 3:35
Ramesh Dharan
32016
edited Nov 30 '18 at 3:35
Ramesh Dharan
32016
edited Nov 30 '18 at 3:35
Ramesh Dharan
32016
32016
asked Nov 23 '18 at 7:02
pavanpavan
1
asked Nov 23 '18 at 7:02
pavanpavan
1
asked Nov 23 '18 at 7:02
pavanpavan
1
1
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
The name you pass to the setuptools.setup function is the name of your package, and its dependencies should be specified in the argument install_requires. I would imagine it works with the DirectRunner because the package is installed on your local machine.
The Beam juliaset example provides a sample setup.py file:
REQUIRED_PACKAGES = ['numpy']
setuptools.setup(
name='juliaset', # this is their package name
version='0.0.1',
description='Julia set workflow package.',
install_requires=REQUIRED_PACKAGES,
...)
PyPI dependencies
If lxml is your only dependency, or all your dependencies are on PyPI, you should be able to use the much simpler requirements.txt file. In general, the setup.py approach requires much more boilerplate.
To use requirements.txt, freeze your dependencies:
pip freeze > requirements.txt
And pass the requirements.txt file to your pipeline:
--requirements_file requirements.txt
See also the Beam documentation's page for various dependency patterns for Python.
answered Nov 28 '18 at 22:02
jagthebeetlejagthebeetle
44858
Thanks for answering my query. I realized my mistake and used requirements file later. And its working as expected.
– pavan
Nov 30 '18 at 4:47
Glad the issue is resolved, please consider to accept the answer.
– MonicaPC
Dec 1 '18 at 0:40
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53442013%2fgcp-dataflowrunner-importerrors%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The name you pass to the setuptools.setup function is the name of your package, and its dependencies should be specified in the argument install_requires. I would imagine it works with the DirectRunner because the package is installed on your local machine.
The Beam juliaset example provides a sample setup.py file:
REQUIRED_PACKAGES = ['numpy']
setuptools.setup(
name='juliaset', # this is their package name
version='0.0.1',
description='Julia set workflow package.',
install_requires=REQUIRED_PACKAGES,
...)
PyPI dependencies
If lxml is your only dependency, or all your dependencies are on PyPI, you should be able to use the much simpler requirements.txt file. In general, the setup.py approach requires much more boilerplate.
To use requirements.txt, freeze your dependencies:
pip freeze > requirements.txt
And pass the requirements.txt file to your pipeline:
--requirements_file requirements.txt
See also the Beam documentation's page for various dependency patterns for Python.
answered Nov 28 '18 at 22:02
jagthebeetlejagthebeetle
44858
Thanks for answering my query. I realized my mistake and used requirements file later. And its working as expected.
– pavan
Nov 30 '18 at 4:47
Glad the issue is resolved, please consider to accept the answer.
– MonicaPC
Dec 1 '18 at 0:40
add a comment |
The name you pass to the setuptools.setup function is the name of your package, and its dependencies should be specified in the argument install_requires. I would imagine it works with the DirectRunner because the package is installed on your local machine.
The Beam juliaset example provides a sample setup.py file:
REQUIRED_PACKAGES = ['numpy']
setuptools.setup(
name='juliaset', # this is their package name
version='0.0.1',
description='Julia set workflow package.',
install_requires=REQUIRED_PACKAGES,
...)
PyPI dependencies
If lxml is your only dependency, or all your dependencies are on PyPI, you should be able to use the much simpler requirements.txt file. In general, the setup.py approach requires much more boilerplate.
To use requirements.txt, freeze your dependencies:
pip freeze > requirements.txt
And pass the requirements.txt file to your pipeline:
--requirements_file requirements.txt
See also the Beam documentation's page for various dependency patterns for Python.
answered Nov 28 '18 at 22:02
jagthebeetlejagthebeetle
44858
Thanks for answering my query. I realized my mistake and used requirements file later. And its working as expected.
– pavan
Nov 30 '18 at 4:47
Glad the issue is resolved, please consider to accept the answer.
– MonicaPC
Dec 1 '18 at 0:40
add a comment |
The name you pass to the setuptools.setup function is the name of your package, and its dependencies should be specified in the argument install_requires. I would imagine it works with the DirectRunner because the package is installed on your local machine.
The Beam juliaset example provides a sample setup.py file:
REQUIRED_PACKAGES = ['numpy']
setuptools.setup(
name='juliaset', # this is their package name
version='0.0.1',
description='Julia set workflow package.',
install_requires=REQUIRED_PACKAGES,
...)
PyPI dependencies
If lxml is your only dependency, or all your dependencies are on PyPI, you should be able to use the much simpler requirements.txt file. In general, the setup.py approach requires much more boilerplate.
To use requirements.txt, freeze your dependencies:
pip freeze > requirements.txt
And pass the requirements.txt file to your pipeline:
--requirements_file requirements.txt
See also the Beam documentation's page for various dependency patterns for Python.
answered Nov 28 '18 at 22:02
jagthebeetlejagthebeetle
44858
The name you pass to the setuptools.setup function is the name of your package, and its dependencies should be specified in the argument install_requires. I would imagine it works with the DirectRunner because the package is installed on your local machine.
The Beam juliaset example provides a sample setup.py file:
REQUIRED_PACKAGES = ['numpy']
setuptools.setup(
name='juliaset', # this is their package name
version='0.0.1',
description='Julia set workflow package.',
install_requires=REQUIRED_PACKAGES,
...)
PyPI dependencies
If lxml is your only dependency, or all your dependencies are on PyPI, you should be able to use the much simpler requirements.txt file. In general, the setup.py approach requires much more boilerplate.
To use requirements.txt, freeze your dependencies:
pip freeze > requirements.txt
And pass the requirements.txt file to your pipeline:
--requirements_file requirements.txt
See also the Beam documentation's page for various dependency patterns for Python.
answered Nov 28 '18 at 22:02
jagthebeetlejagthebeetle
44858
answered Nov 28 '18 at 22:02
jagthebeetlejagthebeetle
44858
answered Nov 28 '18 at 22:02
jagthebeetlejagthebeetle
44858
answered Nov 28 '18 at 22:02
jagthebeetlejagthebeetle
44858
44858
Thanks for answering my query. I realized my mistake and used requirements file later. And its working as expected.
– pavan
Nov 30 '18 at 4:47
Glad the issue is resolved, please consider to accept the answer.
– MonicaPC
Dec 1 '18 at 0:40
add a comment |
Thanks for answering my query. I realized my mistake and used requirements file later. And its working as expected.
– pavan
Nov 30 '18 at 4:47
Glad the issue is resolved, please consider to accept the answer.
– MonicaPC
Dec 1 '18 at 0:40
Thanks for answering my query. I realized my mistake and used requirements file later. And its working as expected.
– pavan
Nov 30 '18 at 4:47
Thanks for answering my query. I realized my mistake and used requirements file later. And its working as expected.
– pavan
Nov 30 '18 at 4:47
Glad the issue is resolved, please consider to accept the answer.
– MonicaPC
Dec 1 '18 at 0:40
Glad the issue is resolved, please consider to accept the answer.
– MonicaPC
Dec 1 '18 at 0:40
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53442013%2fgcp-dataflowrunner-importerrors%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
