Maven conflict in Java app with google-cloud-core-grpc dependency

up vote
6
down vote

favorite

(I've also raised a GitHub issue for this -
https://github.com/googleapis/google-cloud-java/issues/4095)

I have the latest versions of the following 2 dependencies for Apache Beam:

Dependency 1 - google-cloud-dataflow-java-sdk-all

(A distribution of Apache Beam designed to simplify usage of Apache Beam on Google Cloud Dataflow service - https://mvnrepository.com/artifact/com.google.cloud.dataflow/google-cloud-dataflow-java-sdk-all)

<dependency>

  <groupId>com.google.cloud.dataflow</groupId>

  <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>

  <version>2.5.0</version>

</dependency>

Dependency 2 - beam-runners-google-cloud-dataflow-java

(I'm guessing that this allows the actual running of a Beam pipeline inside a Google Cloud Dataflow)

https://mvnrepository.com/artifact/org.apache.beam/beam-runners-google-cloud-dataflow-java

<dependency>

  <groupId>org.apache.beam</groupId>

  <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>

  <version>2.8.0</version>

</dependency>

The Maven install works fine with these 2 dependencies. I need to add the following (3rd) dependency for independently subscribing to a Google Cloud pub/sub outside of an Apache Beam pipeline:

google-cloud-pubsub

https://mvnrepository.com/artifact/com.google.cloud/google-cloud-pubsub

<dependency>

  <groupId>com.google.cloud</groupId>

  <artifactId>google-cloud-pubsub</artifactId>

  <version>1.53.0</version>

</dependency>

(This is the latest version). After this is added I get the following conflict after I do mvn clean install:

Could not resolve version conflict among [com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-protobuf-lite:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-netty-shaded:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-stub:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-auth:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-core:jar:1.2.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-netty:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 -> io.grpc:grpc-core:jar:1.5.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.grpc:grpc-core:jar:1.7.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 -> io.grpc:grpc-core:jar:1.6.1, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-okhttp:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-protobuf-nano:jar:1.2.0 -> io.grpc:grpc-core:jar:1.2.0

I don't get any Maven install conflict if I remove the Apache Beam dependencies and keep the google-cloud-pubsub dependency.

There's a separate Google 'BOM' (Bill of Materials) dependency manager that's supposed to manage sub-dependencies of Google Cloud dependencies, but this doesn't solve the conflict for me:

  <dependencyManagement>

    <dependencies>

      <dependency>

        <groupId>com.google.cloud</groupId>

        <artifactId>google-cloud-bom</artifactId>

        <version>0.71.0-alpha</version>

        <type>pom</type>

        <scope>import</scope>

       </dependency>

     </dependencies>

  </dependencyManagement>

I followed the answer to this question -

How do I resolve a dependency conflict in Maven?

and I identified that

com.google.cloud:google-cloud-pubsub:jar:1.53.0

has a sub-dependency on:

io.grpc:grpc-core:jar:1.16.1

com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0

has sub-dependencies on various versions of:

io.grpc:grpc-core:jar

(GRPC is A high performance, open-source universal RPC framework)

If I add io.grpc:grpc-core:jar:1.6.1 as an explicit dependency to my POM file, it doesn't 'pin' the dependency as it still gets pulled in as a sub-dependency by com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0.

On the following GitHub issue, garrettjonesgoogle confirms on 21 Feb that google-cloud-bom doesn't pin the version for io.grpc:grpc-core:

https://github.com/googleapis/google-cloud-java/issues/2890

It seems that the Apache Beam dependency manages the varying versions of io.grpc:grpc-core:jar internally, however this conflicts with the (consistent) version used by the Google pub/sub dependency.

Is there anything I can do to work around this? Can I maybe isolate sub-dependencies somehow using Maven?

edited Nov 25 at 23:28

asked Nov 22 at 5:15

Chris Halcrow

10.2k46186

Any luck? I'm having the exact same issue at this moment.
– Iso
Nov 22 at 19:40

@Iso I've added more info but still haven't solved my issue. You can look at the answer to this question which may help you identify what the problem dependency is - stackoverflow.com/questions/37594922/…. If you follow the directions, you should get a text file showing dependencies, comma separated by package. Look for different version numbers of the same sub-dependency across 2 or more different packages. Any package you can see duplicated with different version numbers may be your problem package.
– Chris Halcrow
Nov 23 at 0:41

@Iso you could also try this to identify your conflicting dependency, although it didn't work for me it just gave me the same error output that I'm already getting - maven.apache.org/plugins/maven-dependency-plugin/examples/…
– Chris Halcrow
Nov 23 at 2:32

1

@Iso I've added an 'answer'. It's not an actual solution but it identifies the cause of my problem. I'm working around this for now. Can you do the same by pollling instead of using pub/sub?
– Chris Halcrow
Nov 25 at 22:30

add a comment |

up vote
6
down vote

favorite

(I've also raised a GitHub issue for this -
https://github.com/googleapis/google-cloud-java/issues/4095)

I have the latest versions of the following 2 dependencies for Apache Beam:

Dependency 1 - google-cloud-dataflow-java-sdk-all

<dependency>

  <groupId>com.google.cloud.dataflow</groupId>

  <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>

  <version>2.5.0</version>

</dependency>

Dependency 2 - beam-runners-google-cloud-dataflow-java

(I'm guessing that this allows the actual running of a Beam pipeline inside a Google Cloud Dataflow)

https://mvnrepository.com/artifact/org.apache.beam/beam-runners-google-cloud-dataflow-java

<dependency>

  <groupId>org.apache.beam</groupId>

  <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>

  <version>2.8.0</version>

</dependency>

The Maven install works fine with these 2 dependencies. I need to add the following (3rd) dependency for independently subscribing to a Google Cloud pub/sub outside of an Apache Beam pipeline:

google-cloud-pubsub

https://mvnrepository.com/artifact/com.google.cloud/google-cloud-pubsub

<dependency>

  <groupId>com.google.cloud</groupId>

  <artifactId>google-cloud-pubsub</artifactId>

  <version>1.53.0</version>

</dependency>

(This is the latest version). After this is added I get the following conflict after I do mvn clean install:

Could not resolve version conflict among [com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-protobuf-lite:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-netty-shaded:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-stub:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-auth:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-core:jar:1.2.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-netty:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 -> io.grpc:grpc-core:jar:1.5.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.grpc:grpc-core:jar:1.7.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 -> io.grpc:grpc-core:jar:1.6.1, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-okhttp:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-protobuf-nano:jar:1.2.0 -> io.grpc:grpc-core:jar:1.2.0

I don't get any Maven install conflict if I remove the Apache Beam dependencies and keep the google-cloud-pubsub dependency.

There's a separate Google 'BOM' (Bill of Materials) dependency manager that's supposed to manage sub-dependencies of Google Cloud dependencies, but this doesn't solve the conflict for me:

  <dependencyManagement>

    <dependencies>

      <dependency>

        <groupId>com.google.cloud</groupId>

        <artifactId>google-cloud-bom</artifactId>

        <version>0.71.0-alpha</version>

        <type>pom</type>

        <scope>import</scope>

       </dependency>

     </dependencies>

  </dependencyManagement>

I followed the answer to this question -

How do I resolve a dependency conflict in Maven?

and I identified that

com.google.cloud:google-cloud-pubsub:jar:1.53.0

has a sub-dependency on:

io.grpc:grpc-core:jar:1.16.1

com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0

has sub-dependencies on various versions of:

io.grpc:grpc-core:jar

(GRPC is A high performance, open-source universal RPC framework)

On the following GitHub issue, garrettjonesgoogle confirms on 21 Feb that google-cloud-bom doesn't pin the version for io.grpc:grpc-core:

https://github.com/googleapis/google-cloud-java/issues/2890

Is there anything I can do to work around this? Can I maybe isolate sub-dependencies somehow using Maven?

edited Nov 25 at 23:28

asked Nov 22 at 5:15

Chris Halcrow

10.2k46186

Any luck? I'm having the exact same issue at this moment.
– Iso
Nov 22 at 19:40

@Iso I've added more info but still haven't solved my issue. You can look at the answer to this question which may help you identify what the problem dependency is - stackoverflow.com/questions/37594922/…. If you follow the directions, you should get a text file showing dependencies, comma separated by package. Look for different version numbers of the same sub-dependency across 2 or more different packages. Any package you can see duplicated with different version numbers may be your problem package.
– Chris Halcrow
Nov 23 at 0:41

@Iso you could also try this to identify your conflicting dependency, although it didn't work for me it just gave me the same error output that I'm already getting - maven.apache.org/plugins/maven-dependency-plugin/examples/…
– Chris Halcrow
Nov 23 at 2:32

1

@Iso I've added an 'answer'. It's not an actual solution but it identifies the cause of my problem. I'm working around this for now. Can you do the same by pollling instead of using pub/sub?
– Chris Halcrow
Nov 25 at 22:30

add a comment |

up vote
6
down vote

favorite

(I've also raised a GitHub issue for this -
https://github.com/googleapis/google-cloud-java/issues/4095)

I have the latest versions of the following 2 dependencies for Apache Beam:

Dependency 1 - google-cloud-dataflow-java-sdk-all

<dependency>

  <groupId>com.google.cloud.dataflow</groupId>

  <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>

  <version>2.5.0</version>

</dependency>

Dependency 2 - beam-runners-google-cloud-dataflow-java

(I'm guessing that this allows the actual running of a Beam pipeline inside a Google Cloud Dataflow)

https://mvnrepository.com/artifact/org.apache.beam/beam-runners-google-cloud-dataflow-java

<dependency>

  <groupId>org.apache.beam</groupId>

  <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>

  <version>2.8.0</version>

</dependency>

The Maven install works fine with these 2 dependencies. I need to add the following (3rd) dependency for independently subscribing to a Google Cloud pub/sub outside of an Apache Beam pipeline:

google-cloud-pubsub

https://mvnrepository.com/artifact/com.google.cloud/google-cloud-pubsub

<dependency>

  <groupId>com.google.cloud</groupId>

  <artifactId>google-cloud-pubsub</artifactId>

  <version>1.53.0</version>

</dependency>

(This is the latest version). After this is added I get the following conflict after I do mvn clean install:

Could not resolve version conflict among [com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-protobuf-lite:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-netty-shaded:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-stub:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-auth:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-core:jar:1.2.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-netty:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 -> io.grpc:grpc-core:jar:1.5.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.grpc:grpc-core:jar:1.7.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 -> io.grpc:grpc-core:jar:1.6.1, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-okhttp:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-protobuf-nano:jar:1.2.0 -> io.grpc:grpc-core:jar:1.2.0

I don't get any Maven install conflict if I remove the Apache Beam dependencies and keep the google-cloud-pubsub dependency.

There's a separate Google 'BOM' (Bill of Materials) dependency manager that's supposed to manage sub-dependencies of Google Cloud dependencies, but this doesn't solve the conflict for me:

  <dependencyManagement>

    <dependencies>

      <dependency>

        <groupId>com.google.cloud</groupId>

        <artifactId>google-cloud-bom</artifactId>

        <version>0.71.0-alpha</version>

        <type>pom</type>

        <scope>import</scope>

       </dependency>

     </dependencies>

  </dependencyManagement>

I followed the answer to this question -

How do I resolve a dependency conflict in Maven?

and I identified that

com.google.cloud:google-cloud-pubsub:jar:1.53.0

has a sub-dependency on:

io.grpc:grpc-core:jar:1.16.1

com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0

has sub-dependencies on various versions of:

io.grpc:grpc-core:jar

(GRPC is A high performance, open-source universal RPC framework)

On the following GitHub issue, garrettjonesgoogle confirms on 21 Feb that google-cloud-bom doesn't pin the version for io.grpc:grpc-core:

https://github.com/googleapis/google-cloud-java/issues/2890

Is there anything I can do to work around this? Can I maybe isolate sub-dependencies somehow using Maven?

edited Nov 25 at 23:28

asked Nov 22 at 5:15

Chris Halcrow

10.2k46186

(I've also raised a GitHub issue for this -
https://github.com/googleapis/google-cloud-java/issues/4095)

I have the latest versions of the following 2 dependencies for Apache Beam:

Dependency 1 - google-cloud-dataflow-java-sdk-all

<dependency>

  <groupId>com.google.cloud.dataflow</groupId>

  <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>

  <version>2.5.0</version>

</dependency>

Dependency 2 - beam-runners-google-cloud-dataflow-java

(I'm guessing that this allows the actual running of a Beam pipeline inside a Google Cloud Dataflow)

https://mvnrepository.com/artifact/org.apache.beam/beam-runners-google-cloud-dataflow-java

<dependency>

  <groupId>org.apache.beam</groupId>

  <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>

  <version>2.8.0</version>

</dependency>

The Maven install works fine with these 2 dependencies. I need to add the following (3rd) dependency for independently subscribing to a Google Cloud pub/sub outside of an Apache Beam pipeline:

google-cloud-pubsub

https://mvnrepository.com/artifact/com.google.cloud/google-cloud-pubsub

<dependency>

  <groupId>com.google.cloud</groupId>

  <artifactId>google-cloud-pubsub</artifactId>

  <version>1.53.0</version>

</dependency>

(This is the latest version). After this is added I get the following conflict after I do mvn clean install:

Could not resolve version conflict among [com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-protobuf-lite:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-netty-shaded:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-stub:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-auth:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-core:jar:1.2.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-netty:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 -> io.grpc:grpc-core:jar:1.5.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.grpc:grpc-core:jar:1.7.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 -> io.grpc:grpc-core:jar:1.6.1, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-okhttp:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-protobuf-nano:jar:1.2.0 -> io.grpc:grpc-core:jar:1.2.0

I don't get any Maven install conflict if I remove the Apache Beam dependencies and keep the google-cloud-pubsub dependency.

There's a separate Google 'BOM' (Bill of Materials) dependency manager that's supposed to manage sub-dependencies of Google Cloud dependencies, but this doesn't solve the conflict for me:

  <dependencyManagement>

    <dependencies>

      <dependency>

        <groupId>com.google.cloud</groupId>

        <artifactId>google-cloud-bom</artifactId>

        <version>0.71.0-alpha</version>

        <type>pom</type>

        <scope>import</scope>

       </dependency>

     </dependencies>

  </dependencyManagement>

I followed the answer to this question -

How do I resolve a dependency conflict in Maven?

and I identified that

com.google.cloud:google-cloud-pubsub:jar:1.53.0

has a sub-dependency on:

io.grpc:grpc-core:jar:1.16.1

com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0

has sub-dependencies on various versions of:

io.grpc:grpc-core:jar

(GRPC is A high performance, open-source universal RPC framework)

On the following GitHub issue, garrettjonesgoogle confirms on 21 Feb that google-cloud-bom doesn't pin the version for io.grpc:grpc-core:

https://github.com/googleapis/google-cloud-java/issues/2890

Is there anything I can do to work around this? Can I maybe isolate sub-dependencies somehow using Maven?

java maven google-cloud-platform google-cloud-dataflow apache-beam

edited Nov 25 at 23:28

asked Nov 22 at 5:15

Chris Halcrow

10.2k46186

edited Nov 25 at 23:28

asked Nov 22 at 5:15

Chris Halcrow

10.2k46186

edited Nov 25 at 23:28

asked Nov 22 at 5:15

Chris Halcrow

10.2k46186

asked Nov 22 at 5:15

Chris Halcrow

10.2k46186

asked Nov 22 at 5:15

Chris Halcrow

10.2k46186

Any luck? I'm having the exact same issue at this moment.
– Iso
Nov 22 at 19:40

@Iso I've added more info but still haven't solved my issue. You can look at the answer to this question which may help you identify what the problem dependency is - stackoverflow.com/questions/37594922/…. If you follow the directions, you should get a text file showing dependencies, comma separated by package. Look for different version numbers of the same sub-dependency across 2 or more different packages. Any package you can see duplicated with different version numbers may be your problem package.
– Chris Halcrow
Nov 23 at 0:41

@Iso you could also try this to identify your conflicting dependency, although it didn't work for me it just gave me the same error output that I'm already getting - maven.apache.org/plugins/maven-dependency-plugin/examples/…
– Chris Halcrow
Nov 23 at 2:32

1

@Iso I've added an 'answer'. It's not an actual solution but it identifies the cause of my problem. I'm working around this for now. Can you do the same by pollling instead of using pub/sub?
– Chris Halcrow
Nov 25 at 22:30

add a comment |

Any luck? I'm having the exact same issue at this moment.
– Iso
Nov 22 at 19:40

@Iso I've added more info but still haven't solved my issue. You can look at the answer to this question which may help you identify what the problem dependency is - stackoverflow.com/questions/37594922/…. If you follow the directions, you should get a text file showing dependencies, comma separated by package. Look for different version numbers of the same sub-dependency across 2 or more different packages. Any package you can see duplicated with different version numbers may be your problem package.
– Chris Halcrow
Nov 23 at 0:41

@Iso you could also try this to identify your conflicting dependency, although it didn't work for me it just gave me the same error output that I'm already getting - maven.apache.org/plugins/maven-dependency-plugin/examples/…
– Chris Halcrow
Nov 23 at 2:32

1

@Iso I've added an 'answer'. It's not an actual solution but it identifies the cause of my problem. I'm working around this for now. Can you do the same by pollling instead of using pub/sub?
– Chris Halcrow
Nov 25 at 22:30

Any luck? I'm having the exact same issue at this moment.
– Iso
Nov 22 at 19:40

@Iso I've added more info but still haven't solved my issue. You can look at the answer to this question which may help you identify what the problem dependency is - stackoverflow.com/questions/37594922/…. If you follow the directions, you should get a text file showing dependencies, comma separated by package. Look for different version numbers of the same sub-dependency across 2 or more different packages. Any package you can see duplicated with different version numbers may be your problem package.
– Chris Halcrow
Nov 23 at 0:41

@Iso you could also try this to identify your conflicting dependency, although it didn't work for me it just gave me the same error output that I'm already getting - maven.apache.org/plugins/maven-dependency-plugin/examples/…
– Chris Halcrow
Nov 23 at 2:32

@Iso I've added an 'answer'. It's not an actual solution but it identifies the cause of my problem. I'm working around this for now. Can you do the same by pollling instead of using pub/sub?
– Chris Halcrow
Nov 25 at 22:30

add a comment |

2 Answers
2

active

oldest

votes

up vote
3
down vote

I was able to workaround this issue buy just excluding grpc-core from google-cloud-pubsub artifact. It gets incorporated in the dependency tree afterwards with the same version 1.13.1 as a dependency of beam-runners-google-cloud-dataflow-java

This is the exceprt of the pom.xml which worked for me.

<dependencies>

    <dependency>

        <groupId>com.google.cloud</groupId>

        <artifactId>google-cloud-pubsub</artifactId>

        <version>1.53.0</version>

        <exclusions>

            <exclusion>

                <groupId>io.grpc</groupId>

                <artifactId>grpc-core</artifactId>

            </exclusion>

        </exclusions>

    </dependency>

    <dependency>

        <groupId>org.apache.beam</groupId>

        <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>

        <version>2.8.0</version>

    </dependency>

</dependencies>

answered Nov 30 at 13:40

Serg M Ten

3,36011432

add a comment |

up vote
2
down vote

This isn't really a solution however I did identify that the issue is mainly with this dependency:

<dependency>

  <groupId>com.google.cloud.dataflow</groupId>

  <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>

  <version>2.5.0</version>

</dependency>

This artifact has a number of inconsistent dependencies on io.grpc:grpc-core:jar and isolation between the Apache Beam and Google pub sub artifacts aren't really working properly. Whatever version of google-cloud-pubsub is being used, its own dependency on io.grpc:grpc-core:jar will conflict with one of the versions of of io.grpc:grpc-core:jar.

I've raised the following Git issue against the Google artifact:

https://github.com/googleapis/google-cloud-java/issues/4095

And I've raised the following against the Apache Beam artifact (it points back to the Google issue I raised):

https://issues.apache.org/jira/browse/BEAM-6118

I'm addressing the issue by working around it. I can poll the data I need instead of consuming it from a subscription to the pub/sub however this isn't optimal and will lead to significant increased costs on Google Cloud.

edited Nov 25 at 23:10

answered Nov 25 at 22:30

Chris Halcrow

10.2k46186

1

Hey Chris. Excuse me for ignoring your earlier reply. I've got this over-simplified setup for a masterthesis proof-of-concept. I worked around the conflict by separating the projects (as in my case, I was dealing with a bundled Pub/Sub publisher and Dataflow project). This is indeed not optimal, but it will do for the moment being. Thanks for the heads up, i'll keep an eye on the issue.
– Iso
Nov 25 at 22:56

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53424272%2fmaven-conflict-in-java-app-with-google-cloud-core-grpc-dependency%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

2 Answers
2

active

oldest

votes

2 Answers
2

active

oldest

votes

up vote
3
down vote

This is the exceprt of the pom.xml which worked for me.

<dependencies>

    <dependency>

        <groupId>com.google.cloud</groupId>

        <artifactId>google-cloud-pubsub</artifactId>

        <version>1.53.0</version>

        <exclusions>

            <exclusion>

                <groupId>io.grpc</groupId>

                <artifactId>grpc-core</artifactId>

            </exclusion>

        </exclusions>

    </dependency>

    <dependency>

        <groupId>org.apache.beam</groupId>

        <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>

        <version>2.8.0</version>

    </dependency>

</dependencies>

answered Nov 30 at 13:40

Serg M Ten

3,36011432

add a comment |

up vote
3
down vote

This is the exceprt of the pom.xml which worked for me.

<dependencies>

    <dependency>

        <groupId>com.google.cloud</groupId>

        <artifactId>google-cloud-pubsub</artifactId>

        <version>1.53.0</version>

        <exclusions>

            <exclusion>

                <groupId>io.grpc</groupId>

                <artifactId>grpc-core</artifactId>

            </exclusion>

        </exclusions>

    </dependency>

    <dependency>

        <groupId>org.apache.beam</groupId>

        <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>

        <version>2.8.0</version>

    </dependency>

</dependencies>

answered Nov 30 at 13:40

Serg M Ten

3,36011432

add a comment |

up vote
3
down vote

This is the exceprt of the pom.xml which worked for me.

<dependencies>

    <dependency>

        <groupId>com.google.cloud</groupId>

        <artifactId>google-cloud-pubsub</artifactId>

        <version>1.53.0</version>

        <exclusions>

            <exclusion>

                <groupId>io.grpc</groupId>

                <artifactId>grpc-core</artifactId>

            </exclusion>

        </exclusions>

    </dependency>

    <dependency>

        <groupId>org.apache.beam</groupId>

        <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>

        <version>2.8.0</version>

    </dependency>

</dependencies>

answered Nov 30 at 13:40

Serg M Ten

3,36011432

This is the exceprt of the pom.xml which worked for me.

<dependencies>

    <dependency>

        <groupId>com.google.cloud</groupId>

        <artifactId>google-cloud-pubsub</artifactId>

        <version>1.53.0</version>

        <exclusions>

            <exclusion>

                <groupId>io.grpc</groupId>

                <artifactId>grpc-core</artifactId>

            </exclusion>

        </exclusions>

    </dependency>

    <dependency>

        <groupId>org.apache.beam</groupId>

        <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>

        <version>2.8.0</version>

    </dependency>

</dependencies>

answered Nov 30 at 13:40

Serg M Ten

3,36011432

answered Nov 30 at 13:40

Serg M Ten

3,36011432

answered Nov 30 at 13:40

Serg M Ten

3,36011432

answered Nov 30 at 13:40

Serg M Ten

3,36011432

add a comment |

up vote
2
down vote

This isn't really a solution however I did identify that the issue is mainly with this dependency:

<dependency>

  <groupId>com.google.cloud.dataflow</groupId>

  <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>

  <version>2.5.0</version>

</dependency>

I've raised the following Git issue against the Google artifact:

https://github.com/googleapis/google-cloud-java/issues/4095

And I've raised the following against the Apache Beam artifact (it points back to the Google issue I raised):

https://issues.apache.org/jira/browse/BEAM-6118

edited Nov 25 at 23:10

answered Nov 25 at 22:30

Chris Halcrow

10.2k46186

1

Hey Chris. Excuse me for ignoring your earlier reply. I've got this over-simplified setup for a masterthesis proof-of-concept. I worked around the conflict by separating the projects (as in my case, I was dealing with a bundled Pub/Sub publisher and Dataflow project). This is indeed not optimal, but it will do for the moment being. Thanks for the heads up, i'll keep an eye on the issue.
– Iso
Nov 25 at 22:56

add a comment |

up vote
2
down vote

This isn't really a solution however I did identify that the issue is mainly with this dependency:

<dependency>

  <groupId>com.google.cloud.dataflow</groupId>

  <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>

  <version>2.5.0</version>

</dependency>

I've raised the following Git issue against the Google artifact:

https://github.com/googleapis/google-cloud-java/issues/4095

And I've raised the following against the Apache Beam artifact (it points back to the Google issue I raised):

https://issues.apache.org/jira/browse/BEAM-6118

edited Nov 25 at 23:10

answered Nov 25 at 22:30

Chris Halcrow

10.2k46186

1

Hey Chris. Excuse me for ignoring your earlier reply. I've got this over-simplified setup for a masterthesis proof-of-concept. I worked around the conflict by separating the projects (as in my case, I was dealing with a bundled Pub/Sub publisher and Dataflow project). This is indeed not optimal, but it will do for the moment being. Thanks for the heads up, i'll keep an eye on the issue.
– Iso
Nov 25 at 22:56

add a comment |

up vote
2
down vote

This isn't really a solution however I did identify that the issue is mainly with this dependency:

<dependency>

  <groupId>com.google.cloud.dataflow</groupId>

  <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>

  <version>2.5.0</version>

</dependency>

I've raised the following Git issue against the Google artifact:

https://github.com/googleapis/google-cloud-java/issues/4095

And I've raised the following against the Apache Beam artifact (it points back to the Google issue I raised):

https://issues.apache.org/jira/browse/BEAM-6118

edited Nov 25 at 23:10

answered Nov 25 at 22:30

Chris Halcrow

10.2k46186

This isn't really a solution however I did identify that the issue is mainly with this dependency:

<dependency>

  <groupId>com.google.cloud.dataflow</groupId>

  <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>

  <version>2.5.0</version>

</dependency>

I've raised the following Git issue against the Google artifact:

https://github.com/googleapis/google-cloud-java/issues/4095

And I've raised the following against the Apache Beam artifact (it points back to the Google issue I raised):

https://issues.apache.org/jira/browse/BEAM-6118

edited Nov 25 at 23:10

answered Nov 25 at 22:30

Chris Halcrow

10.2k46186

edited Nov 25 at 23:10

answered Nov 25 at 22:30

Chris Halcrow

10.2k46186

answered Nov 25 at 22:30

Chris Halcrow

10.2k46186

answered Nov 25 at 22:30

Chris Halcrow

10.2k46186

1

Hey Chris. Excuse me for ignoring your earlier reply. I've got this over-simplified setup for a masterthesis proof-of-concept. I worked around the conflict by separating the projects (as in my case, I was dealing with a bundled Pub/Sub publisher and Dataflow project). This is indeed not optimal, but it will do for the moment being. Thanks for the heads up, i'll keep an eye on the issue.
– Iso
Nov 25 at 22:56

add a comment |

1

Hey Chris. Excuse me for ignoring your earlier reply. I've got this over-simplified setup for a masterthesis proof-of-concept. I worked around the conflict by separating the projects (as in my case, I was dealing with a bundled Pub/Sub publisher and Dataflow project). This is indeed not optimal, but it will do for the moment being. Thanks for the heads up, i'll keep an eye on the issue.
– Iso
Nov 25 at 22:56

Hey Chris. Excuse me for ignoring your earlier reply. I've got this over-simplified setup for a masterthesis proof-of-concept. I worked around the conflict by separating the projects (as in my case, I was dealing with a bundled Pub/Sub publisher and Dataflow project). This is indeed not optimal, but it will do for the moment being. Thanks for the heads up, i'll keep an eye on the issue.
– Iso
Nov 25 at 22:56

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

Some of your past answers have not been well-received, and you're in danger of being blocked from answering.

Please pay close attention to the following guidance:

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Htykuut