Fetching data from QueryDatabaseTable in NiFi

up vote
1
down vote

favorite

I have a Database in MySQL that has almost 2 million row count and 30 columns. My flow is QueryDatabaseTable -> convertrecord -> PutElasticSearchHttpRecord. My QueryDatabaseTable has following configs (please see the screenshots):

enter image description here

My Jjava.args values are as follows: (Although I updated the values of following two as 8 GB, still I am getting java heap space error)

java.arg.2=-Xms512m
java.arg.3=-Xmx512m

I am applying select * from Tablename (will result in 749,748 rows across 9 columns) but getting "Java heap space Error". Is it because i am trying to call 749,748 rows across 9 columns?

What will be the best way to call such huge dataset from MySQL using NiFi QueryDatabaseTable processor. Any help is much appreciated. Thank You!

edited Nov 25 at 21:41

marc_s

568k12810991249

asked Nov 21 at 19:47

shrads

1167

Thanks for updating the question, after updating jvm heap space to 8GB did you restarted the NiFi?, after restart then only nifi will run with 8GB JVM memory.
– Shu
Nov 21 at 21:22

Did the issue got resolved?.
– Shu
Nov 27 at 15:06

1

@Shu yes Thank you! I didnt stop and restarted the NiFi. Now it is solved :)
– shrads
Nov 27 at 20:04

add a comment |

up vote
1
down vote

favorite

enter image description here

My Jjava.args values are as follows: (Although I updated the values of following two as 8 GB, still I am getting java heap space error)

java.arg.2=-Xms512m
java.arg.3=-Xmx512m

I am applying select * from Tablename (will result in 749,748 rows across 9 columns) but getting "Java heap space Error". Is it because i am trying to call 749,748 rows across 9 columns?

What will be the best way to call such huge dataset from MySQL using NiFi QueryDatabaseTable processor. Any help is much appreciated. Thank You!

edited Nov 25 at 21:41

marc_s

568k12810991249

asked Nov 21 at 19:47

shrads

1167

Thanks for updating the question, after updating jvm heap space to 8GB did you restarted the NiFi?, after restart then only nifi will run with 8GB JVM memory.
– Shu
Nov 21 at 21:22

Did the issue got resolved?.
– Shu
Nov 27 at 15:06

1

@Shu yes Thank you! I didnt stop and restarted the NiFi. Now it is solved :)
– shrads
Nov 27 at 20:04

add a comment |

up vote
1
down vote

favorite

enter image description here

My Jjava.args values are as follows: (Although I updated the values of following two as 8 GB, still I am getting java heap space error)

java.arg.2=-Xms512m
java.arg.3=-Xmx512m

I am applying select * from Tablename (will result in 749,748 rows across 9 columns) but getting "Java heap space Error". Is it because i am trying to call 749,748 rows across 9 columns?

What will be the best way to call such huge dataset from MySQL using NiFi QueryDatabaseTable processor. Any help is much appreciated. Thank You!

edited Nov 25 at 21:41

marc_s

568k12810991249

asked Nov 21 at 19:47

shrads

1167

enter image description here

My Jjava.args values are as follows: (Although I updated the values of following two as 8 GB, still I am getting java heap space error)

java.arg.2=-Xms512m
java.arg.3=-Xmx512m

I am applying select * from Tablename (will result in 749,748 rows across 9 columns) but getting "Java heap space Error". Is it because i am trying to call 749,748 rows across 9 columns?

What will be the best way to call such huge dataset from MySQL using NiFi QueryDatabaseTable processor. Any help is much appreciated. Thank You!

mysql apache apache-nifi

edited Nov 25 at 21:41

marc_s

568k12810991249

asked Nov 21 at 19:47

shrads

1167

edited Nov 25 at 21:41

marc_s

568k12810991249

asked Nov 21 at 19:47

shrads

1167

edited Nov 25 at 21:41

marc_s

568k12810991249

edited Nov 25 at 21:41

marc_s

568k12810991249

edited Nov 25 at 21:41

marc_s

568k12810991249

asked Nov 21 at 19:47

shrads

1167

asked Nov 21 at 19:47

shrads

1167

asked Nov 21 at 19:47

shrads

1167

Thanks for updating the question, after updating jvm heap space to 8GB did you restarted the NiFi?, after restart then only nifi will run with 8GB JVM memory.
– Shu
Nov 21 at 21:22

Did the issue got resolved?.
– Shu
Nov 27 at 15:06

1

@Shu yes Thank you! I didnt stop and restarted the NiFi. Now it is solved :)
– shrads
Nov 27 at 20:04

add a comment |

Thanks for updating the question, after updating jvm heap space to 8GB did you restarted the NiFi?, after restart then only nifi will run with 8GB JVM memory.
– Shu
Nov 21 at 21:22

Did the issue got resolved?.
– Shu
Nov 27 at 15:06

1

@Shu yes Thank you! I didnt stop and restarted the NiFi. Now it is solved :)
– shrads
Nov 27 at 20:04

Thanks for updating the question, after updating jvm heap space to 8GB did you restarted the NiFi?, after restart then only nifi will run with 8GB JVM memory.
– Shu
Nov 21 at 21:22

Did the issue got resolved?.
– Shu
Nov 27 at 15:06

@Shu yes Thank you! I didnt stop and restarted the NiFi. Now it is solved :)
– shrads
Nov 27 at 20:04

add a comment |

1 Answer
1

active

oldest

votes

up vote
0
down vote

accepted

As discussed in above comments.

NiFi default JVM configs are java.arg.2=-Xms512m java.arg.3=-Xmx512m and this is root cause for the outofmemory issue.

To resolve this issue increase the the heap space in $NIFI_HOME/conf/bootstrap.conf file

As per your requirement change the configs as shown below i have increased the heap space to 8GB

# JVM memory settings

java.arg.2=-Xms8912m

java.arg.3=-Xmx8912m

And to get these configs take into effect in your NiFi instance, RESTART the whole NiFi instance, Then only the NiFi instance will run with the configured heap space.

For more details regards to JVM outofmemory issues refer to this link.

answered Nov 28 at 0:09

Shu

4,1562418

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53419517%2ffetching-data-from-querydatabasetable-in-nifi%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

up vote
0
down vote

accepted

As discussed in above comments.

NiFi default JVM configs are java.arg.2=-Xms512m java.arg.3=-Xmx512m and this is root cause for the outofmemory issue.

To resolve this issue increase the the heap space in $NIFI_HOME/conf/bootstrap.conf file

As per your requirement change the configs as shown below i have increased the heap space to 8GB

# JVM memory settings

java.arg.2=-Xms8912m

java.arg.3=-Xmx8912m

And to get these configs take into effect in your NiFi instance, RESTART the whole NiFi instance, Then only the NiFi instance will run with the configured heap space.

For more details regards to JVM outofmemory issues refer to this link.

answered Nov 28 at 0:09

Shu

4,1562418

add a comment |

up vote
0
down vote

accepted

As discussed in above comments.

NiFi default JVM configs are java.arg.2=-Xms512m java.arg.3=-Xmx512m and this is root cause for the outofmemory issue.

To resolve this issue increase the the heap space in $NIFI_HOME/conf/bootstrap.conf file

As per your requirement change the configs as shown below i have increased the heap space to 8GB

# JVM memory settings

java.arg.2=-Xms8912m

java.arg.3=-Xmx8912m

And to get these configs take into effect in your NiFi instance, RESTART the whole NiFi instance, Then only the NiFi instance will run with the configured heap space.

For more details regards to JVM outofmemory issues refer to this link.

answered Nov 28 at 0:09

Shu

4,1562418

add a comment |

up vote
0
down vote

accepted

As discussed in above comments.

NiFi default JVM configs are java.arg.2=-Xms512m java.arg.3=-Xmx512m and this is root cause for the outofmemory issue.

To resolve this issue increase the the heap space in $NIFI_HOME/conf/bootstrap.conf file

As per your requirement change the configs as shown below i have increased the heap space to 8GB

# JVM memory settings

java.arg.2=-Xms8912m

java.arg.3=-Xmx8912m

And to get these configs take into effect in your NiFi instance, RESTART the whole NiFi instance, Then only the NiFi instance will run with the configured heap space.

For more details regards to JVM outofmemory issues refer to this link.

answered Nov 28 at 0:09

Shu

4,1562418

As discussed in above comments.

NiFi default JVM configs are java.arg.2=-Xms512m java.arg.3=-Xmx512m and this is root cause for the outofmemory issue.

To resolve this issue increase the the heap space in $NIFI_HOME/conf/bootstrap.conf file

As per your requirement change the configs as shown below i have increased the heap space to 8GB

# JVM memory settings

java.arg.2=-Xms8912m

java.arg.3=-Xmx8912m

And to get these configs take into effect in your NiFi instance, RESTART the whole NiFi instance, Then only the NiFi instance will run with the configured heap space.

For more details regards to JVM outofmemory issues refer to this link.

answered Nov 28 at 0:09

Shu

4,1562418

answered Nov 28 at 0:09

Shu

4,1562418

answered Nov 28 at 0:09

Shu

4,1562418

answered Nov 28 at 0:09

Shu

4,1562418

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

Some of your past answers have not been well-received, and you're in danger of being blocked from answering.

Please pay close attention to the following guidance:

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Htykuut