Is there a way to connect to multiple databases in multiple hosts using Kafka Connect?
up vote
0
down vote
favorite
I have a need to get data from Informix database using Kafka Connect. The scenario is this - I have 50 Informix Databases residing in 50 hosts. What I have understood by reading from Kafka connect is that we need to install the Kafka connect in each hosts to get the data from the database residing in that host. My question is this - Is there a way in which I can create the connectors centrally for these 50 hosts instead of installing into each of them and pull data from the databases?
apache-kafka informix apache-kafka-connect
edited Nov 21 at 22:57
cricket_007
78k1142108
asked Nov 21 at 18:35
Anish Kurian
53117
add a comment |
up vote
0
down vote
favorite
I have a need to get data from Informix database using Kafka Connect. The scenario is this - I have 50 Informix Databases residing in 50 hosts. What I have understood by reading from Kafka connect is that we need to install the Kafka connect in each hosts to get the data from the database residing in that host. My question is this - Is there a way in which I can create the connectors centrally for these 50 hosts instead of installing into each of them and pull data from the databases?
apache-kafka informix apache-kafka-connect
edited Nov 21 at 22:57
cricket_007
78k1142108
asked Nov 21 at 18:35
Anish Kurian
53117
1
Where did you read that you needed to install Connect on your database hosts?
– Mickael Maison
Nov 21 at 21:09
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have a need to get data from Informix database using Kafka Connect. The scenario is this - I have 50 Informix Databases residing in 50 hosts. What I have understood by reading from Kafka connect is that we need to install the Kafka connect in each hosts to get the data from the database residing in that host. My question is this - Is there a way in which I can create the connectors centrally for these 50 hosts instead of installing into each of them and pull data from the databases?
apache-kafka informix apache-kafka-connect
edited Nov 21 at 22:57
cricket_007
78k1142108
asked Nov 21 at 18:35
Anish Kurian
53117
I have a need to get data from Informix database using Kafka Connect. The scenario is this - I have 50 Informix Databases residing in 50 hosts. What I have understood by reading from Kafka connect is that we need to install the Kafka connect in each hosts to get the data from the database residing in that host. My question is this - Is there a way in which I can create the connectors centrally for these 50 hosts instead of installing into each of them and pull data from the databases?
apache-kafka informix apache-kafka-connect
apache-kafka informix apache-kafka-connect
edited Nov 21 at 22:57
cricket_007
78k1142108
asked Nov 21 at 18:35
Anish Kurian
53117
edited Nov 21 at 22:57
cricket_007
78k1142108
asked Nov 21 at 18:35
Anish Kurian
53117
edited Nov 21 at 22:57
cricket_007
78k1142108
edited Nov 21 at 22:57
cricket_007
78k1142108
edited Nov 21 at 22:57
cricket_007
78k1142108
78k1142108
asked Nov 21 at 18:35
Anish Kurian
53117
asked Nov 21 at 18:35
Anish Kurian
53117
asked Nov 21 at 18:35
Anish Kurian
53117
53117
1
Where did you read that you needed to install Connect on your database hosts?
– Mickael Maison
Nov 21 at 21:09
add a comment |
1
Where did you read that you needed to install Connect on your database hosts?
– Mickael Maison
Nov 21 at 21:09
1
1
Where did you read that you needed to install Connect on your database hosts?
– Mickael Maison
Nov 21 at 21:09
Where did you read that you needed to install Connect on your database hosts?
– Mickael Maison
Nov 21 at 21:09
add a comment |
2 Answers
2
active
oldest
votes
up vote
1
down vote
Kafka Connect JDBC does not have to run on the database, just as other JDBC clients don't, so you can a have a Kafka Connect cluster be larger or smaller than your database pool.
Informix seems to have a thing called "CDC Replication Engine for Kafka", however, which might be something worth looking into, as CDC overall causes less load on the database
answered Nov 21 at 22:56
cricket_007
78k1142108
add a comment |
up vote
0
down vote
You don’t need any additional software installation on the system where Informix server is running.I am not fully clear about the question or the type of operation you are plan to do. If you are planning to setup a real time replication type of scenario, then you may have to invoke CDC API. Then one-time setup of CDC API at server is needed, then this APIs can be invoked using any Informix database driver API. If you are plan to read existing data from table(s) and pump into Kafka topic, then no need of any additional setup at server side. You could connect to all 50 database server(s) from a single program (remotely) and then pump those records to the Kafka topic(s). Base on the program language you are using you may choose Informix database driver.
answered Nov 27 at 6:06
Satyan
23614
add a comment |
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
Kafka Connect JDBC does not have to run on the database, just as other JDBC clients don't, so you can a have a Kafka Connect cluster be larger or smaller than your database pool.
Informix seems to have a thing called "CDC Replication Engine for Kafka", however, which might be something worth looking into, as CDC overall causes less load on the database
answered Nov 21 at 22:56
cricket_007
78k1142108
add a comment |
up vote
1
down vote
Kafka Connect JDBC does not have to run on the database, just as other JDBC clients don't, so you can a have a Kafka Connect cluster be larger or smaller than your database pool.
Informix seems to have a thing called "CDC Replication Engine for Kafka", however, which might be something worth looking into, as CDC overall causes less load on the database
answered Nov 21 at 22:56
cricket_007
78k1142108
add a comment |
up vote
1
down vote
up vote
1
down vote
Kafka Connect JDBC does not have to run on the database, just as other JDBC clients don't, so you can a have a Kafka Connect cluster be larger or smaller than your database pool.
Informix seems to have a thing called "CDC Replication Engine for Kafka", however, which might be something worth looking into, as CDC overall causes less load on the database
answered Nov 21 at 22:56
cricket_007
78k1142108
Kafka Connect JDBC does not have to run on the database, just as other JDBC clients don't, so you can a have a Kafka Connect cluster be larger or smaller than your database pool.
Informix seems to have a thing called "CDC Replication Engine for Kafka", however, which might be something worth looking into, as CDC overall causes less load on the database
answered Nov 21 at 22:56
cricket_007
78k1142108
answered Nov 21 at 22:56
cricket_007
78k1142108
answered Nov 21 at 22:56
cricket_007
78k1142108
answered Nov 21 at 22:56
cricket_007
78k1142108
78k1142108
add a comment |
add a comment |
up vote
0
down vote
You don’t need any additional software installation on the system where Informix server is running.I am not fully clear about the question or the type of operation you are plan to do. If you are planning to setup a real time replication type of scenario, then you may have to invoke CDC API. Then one-time setup of CDC API at server is needed, then this APIs can be invoked using any Informix database driver API. If you are plan to read existing data from table(s) and pump into Kafka topic, then no need of any additional setup at server side. You could connect to all 50 database server(s) from a single program (remotely) and then pump those records to the Kafka topic(s). Base on the program language you are using you may choose Informix database driver.
answered Nov 27 at 6:06
Satyan
23614
add a comment |
up vote
0
down vote
You don’t need any additional software installation on the system where Informix server is running.I am not fully clear about the question or the type of operation you are plan to do. If you are planning to setup a real time replication type of scenario, then you may have to invoke CDC API. Then one-time setup of CDC API at server is needed, then this APIs can be invoked using any Informix database driver API. If you are plan to read existing data from table(s) and pump into Kafka topic, then no need of any additional setup at server side. You could connect to all 50 database server(s) from a single program (remotely) and then pump those records to the Kafka topic(s). Base on the program language you are using you may choose Informix database driver.
answered Nov 27 at 6:06
Satyan
23614
add a comment |
up vote
0
down vote
up vote
0
down vote
You don’t need any additional software installation on the system where Informix server is running.I am not fully clear about the question or the type of operation you are plan to do. If you are planning to setup a real time replication type of scenario, then you may have to invoke CDC API. Then one-time setup of CDC API at server is needed, then this APIs can be invoked using any Informix database driver API. If you are plan to read existing data from table(s) and pump into Kafka topic, then no need of any additional setup at server side. You could connect to all 50 database server(s) from a single program (remotely) and then pump those records to the Kafka topic(s). Base on the program language you are using you may choose Informix database driver.
answered Nov 27 at 6:06
Satyan
23614
You don’t need any additional software installation on the system where Informix server is running.I am not fully clear about the question or the type of operation you are plan to do. If you are planning to setup a real time replication type of scenario, then you may have to invoke CDC API. Then one-time setup of CDC API at server is needed, then this APIs can be invoked using any Informix database driver API. If you are plan to read existing data from table(s) and pump into Kafka topic, then no need of any additional setup at server side. You could connect to all 50 database server(s) from a single program (remotely) and then pump those records to the Kafka topic(s). Base on the program language you are using you may choose Informix database driver.
answered Nov 27 at 6:06
Satyan
23614
answered Nov 27 at 6:06
Satyan
23614
answered Nov 27 at 6:06
Satyan
23614
answered Nov 27 at 6:06
Satyan
23614
23614
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53418543%2fis-there-a-way-to-connect-to-multiple-databases-in-multiple-hosts-using-kafka-co%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Where did you read that you needed to install Connect on your database hosts?
– Mickael Maison
Nov 21 at 21:09