Htykuut

(I've also raised a GitHub issue for this -
https://github.com/googleapis/google-cloud-java/issues/4095)

I have the latest versions of the following 2 dependencies for Apache Beam:

Dependency 1 - google-cloud-dataflow-java-sdk-all

(A distribution of Apache Beam designed to simplify usage of Apache Beam on Google Cloud Dataflow service - https://mvnrepository.com/artifact/com.google.cloud.dataflow/google-cloud-dataflow-java-sdk-all)

<dependency>

  <groupId>com.google.cloud.dataflow</groupId>

  <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>

  <version>2.5.0</version>

</dependency>

Dependency 2 - beam-runners-google-cloud-dataflow-java

(I'm guessing that this allows the actual running of a Beam pipeline inside a Google Cloud Dataflow)

https://mvnrepository.com/artifact/org.apache.beam/beam-runners-google-cloud-dataflow-java

<dependency>

  <groupId>org.apache.beam</groupId>

  <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>

  <version>2.8.0</version>

</dependency>

The Maven install works fine with these 2 dependencies. I need to add the following (3rd) dependency for independently subscribing to a Google Cloud pub/sub outside of an Apache Beam pipeline:

google-cloud-pubsub

https://mvnrepository.com/artifact/com.google.cloud/google-cloud-pubsub

<dependency>

  <groupId>com.google.cloud</groupId>

  <artifactId>google-cloud-pubsub</artifactId>

  <version>1.53.0</version>

</dependency>

(This is the latest version). After this is added I get the following conflict after I do mvn clean install:

Could not resolve version conflict among [com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-protobuf-lite:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-netty-shaded:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-stub:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-auth:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-core:jar:1.2.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-netty:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 -> io.grpc:grpc-core:jar:1.5.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.grpc:grpc-core:jar:1.7.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 -> io.grpc:grpc-core:jar:1.6.1, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-okhttp:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-protobuf-nano:jar:1.2.0 -> io.grpc:grpc-core:jar:1.2.0

I don't get any Maven install conflict if I remove the Apache Beam dependencies and keep the google-cloud-pubsub dependency.

There's a separate Google 'BOM' (Bill of Materials) dependency manager that's supposed to manage sub-dependencies of Google Cloud dependencies, but this doesn't solve the conflict for me:

  <dependencyManagement>

    <dependencies>

      <dependency>

        <groupId>com.google.cloud</groupId>

        <artifactId>google-cloud-bom</artifactId>

        <version>0.71.0-alpha</version>

        <type>pom</type>

        <scope>import</scope>

       </dependency>

     </dependencies>

  </dependencyManagement>

I followed the answer to this question -

How do I resolve a dependency conflict in Maven?

and I identified that

• com.google.cloud:google-cloud-pubsub:jar:1.53.0

has a sub-dependency on:

io.grpc:grpc-core:jar:1.16.1

• com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0

has sub-dependencies on various versions of:

io.grpc:grpc-core:jar

(GRPC is A high performance, open-source universal RPC framework)

If I add io.grpc:grpc-core:jar:1.6.1 as an explicit dependency to my POM file, it doesn't 'pin' the dependency as it still gets pulled in as a sub-dependency by com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0.

On the following GitHub issue, garrettjonesgoogle confirms on 21 Feb that google-cloud-bom doesn't pin the version for io.grpc:grpc-core:

https://github.com/googleapis/google-cloud-java/issues/2890

It seems that the Apache Beam dependency manages the varying versions of io.grpc:grpc-core:jar internally, however this conflicts with the (consistent) version used by the Google pub/sub dependency.

Is there anything I can do to work around this? Can I maybe isolate sub-dependencies somehow using Maven?

I was able to workaround this issue buy just excluding grpc-core from google-cloud-pubsub artifact. It gets incorporated in the dependency tree afterwards with the same version 1.13.1 as a dependency of beam-runners-google-cloud-dataflow-java

This is the exceprt of the pom.xml which worked for me.

<dependencies>

    <dependency>

        <groupId>com.google.cloud</groupId>

        <artifactId>google-cloud-pubsub</artifactId>

        <version>1.53.0</version>

        <exclusions>

            <exclusion>

                <groupId>io.grpc</groupId>

                <artifactId>grpc-core</artifactId>

            </exclusion>

        </exclusions>

    </dependency>

    <dependency>

        <groupId>org.apache.beam</groupId>

        <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>

        <version>2.8.0</version>

    </dependency>

</dependencies>

This isn't really a solution however I did identify that the issue is mainly with this dependency:

<dependency>

  <groupId>com.google.cloud.dataflow</groupId>

  <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>

  <version>2.5.0</version>

</dependency>

This artifact has a number of inconsistent dependencies on io.grpc:grpc-core:jar and isolation between the Apache Beam and Google pub sub artifacts aren't really working properly. Whatever version of google-cloud-pubsub is being used, its own dependency on io.grpc:grpc-core:jar will conflict with one of the versions of of io.grpc:grpc-core:jar.

I've raised the following Git issue against the Google artifact:

https://github.com/googleapis/google-cloud-java/issues/4095

And I've raised the following against the Apache Beam artifact (it points back to the Google issue I raised):

https://issues.apache.org/jira/browse/BEAM-6118

I'm addressing the issue by working around it. I can poll the data I need instead of consuming it from a subscription to the pub/sub however this isn't optimal and will lead to significant increased costs on Google Cloud.