Support for proto files from dependencies (maven)

[raboof]

In SBT we can load .proto files from the project or from a dependency. It would be great if we could also support this from Gradle and Maven.

[patriknw]

Is this needed from MVP? Is it difficult?

[raboof]

Not sure if difficult, removing it from the MVP for now (though we might want/have to do it anyway to support the lagom workflow, that's an unknown for me at this point).

[johanandren]

Did you learn that it was needed for Lagom @raboof given that you put it pack?

[raboof]

I don't remember 😯 ... @renatocaval what do you think, would it be common in Lagom to publish the .proto file in one artifact and generate the client in another, or would it be OK to copy the service definition manually for MVP?

[johanandren]

Isn't Lagom using sbt though?

[raboof]

Also Maven

[pradasouvanlasy]

Hi guys,

while using Maven, an alternative could have been to leverage the maven-dependency-plugin and the unpack goal to extract .proto files from external dependencies and make them available at compile time.
However, I wanted to extract such external files somewhere under the target directory but it seems that the akka-grpc-maven-plugin does not support reading .proto files that are not within the src hierarchy.

The obvious workaround would be to extract those external .proto files somewhere under src to duplicate them into the current module. I don't think this is ideal since it could allow people to make unwanted changes in those external .proto files.
WDYT? Would it make sense to you to allow protoPaths that are pointing somewhere else than src?

EDIT: nevermind, it seems that we must actually provide the full protoPaths such as

<protoPaths>
  <protoPath>target/protobuf_external/common</protoPath>
  <protoPath>target/protobuf_external/common/foo</protoPath>
  <protoPath>src/main/protobuf</protoPath>
</protoPaths>

Simply providing target/protobuf_external does not make it.
Will continue to experiment a bit to see whether I manage to reach an acceptable setup

[raboof]

Will continue to experiment a bit to see whether I manage to reach an acceptable setup

Interesting, looking forward to what you'll find!

[algolov]

Continuing @pradasouvanlasy thought, I came to the following fully working configuration:

<build>
    <plugins>
    	<!--  unpack proto files from external dependencies   -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <executions>
                <execution>
                    <id>unpack</id>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>unpack</goal>
                    </goals>
                    <configuration>
                        <artifactItems>
                            <artifactItem>
                                <groupId>com.some.company</groupId>
                                <artifactId>company-protos</artifactId>
                                <version>${company-protos.version}</version>
                                <type>jar</type>
                                <overWrite>true</overWrite>
                                <outputDirectory>${project.build.directory}/proto</outputDirectory>
                                <includes>**/*.proto</includes>
                                <excludes>**/*.java</excludes>
                            </artifactItem>

                            <!-- to use protobuf common types -->
                            <artifactItem>
                                <groupId>com.google.protobuf</groupId>
                                <artifactId>protobuf-java</artifactId>
                                <version>${protobuf-java.version}</version>
                                <type>jar</type>
                                <overWrite>true</overWrite>
                                <outputDirectory>${project.build.directory}/proto</outputDirectory>
                                <includes>**/*.proto</includes>
                            </artifactItem>

                            <!-- to use protobuf common types -->
                            <artifactItem>
                                <groupId>com.google.api.grpc</groupId>
                                <artifactId>proto-google-common-protos</artifactId>
                                <version>${proto-google-common-protos.version}</version>
                                <type>jar</type>
                                <overWrite>true</overWrite>
                                <outputDirectory>${project.build.directory}/proto</outputDirectory>
                                <includes>**/*.proto</includes>
                            </artifactItem>
                        </artifactItems>
                        <overWriteReleases>false</overWriteReleases>
                        <overWriteSnapshots>true</overWriteSnapshots>
                    </configuration>
                </execution>
            </executions>
        </plugin>

        <plugin>
            <groupId>com.lightbend.akka.grpc</groupId>
            <artifactId>akka-grpc-maven-plugin</artifactId>
            <version>${akka-grpc-maven-plugin.version}</version>
            <configuration>
                <language>Scala</language>
                <generateClient>true</generateClient>
                <generateServer>true</generateServer>
                <protoPaths>
                    <protoPath>target/proto</protoPath>
                </protoPaths>
            </configuration>
            <executions>
                <execution>
                    <id>generate</id>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>generate</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

        <!-- makes akka-grpc-maven-plugin generated classes available as sources -->
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>build-helper-maven-plugin</artifactId>
            <version>${build-helper-maven-plugin}</version>
            <executions>
                <execution>
                    <id>add-source</id>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>add-source</goal>
                    </goals>
                    <configuration>
                        <sources>
                            <source>${project.build.directory}/generated-sources/akka-grpc-scala/</source>
                        </sources>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

[pradasouvanlasy]

@algolov > yes that's more or less what I did but it did not fully work on my end if the external proto files were unpacked somewhere under target (or ${project.build.directory}).
My service proto file imports some of those external proto files and code generation failed since it could not "discover" them properly despite the relevant paths have been declared in <protoPaths>.
I had to unpack the external protos under src to make this work.

If it worked on your end, then that's good news! Will check later what I could have missed. May have goofed at the time

EDIT: did you have any proto file in the project itself, especially the gRPC service file? Or did it come from the company-protos artifact in your example?

[eshepelyuk]

@raboof this is already implemented in protobuf plugin for Gradle.
https://github.com/google/protobuf-gradle-plugin#protos-in-dependencies

[raboof]

We should have split this issue into a gradle one and a maven one ;)

this is already implemented in protobuf plugin for Gradle. https://github.com/google/protobuf-gradle-plugin#protos-in-dependencies

Nice. I guess it does make sense to add a note to our documentation (linking to that page)

[eshepelyuk]

We should have split this issue into a gradle one and a maven one ;)

this is already implemented in protobuf plugin for Gradle. https://github.com/google/protobuf-gradle-plugin#protos-in-dependencies

Nice. I guess it does make sense to add a note to our documentation (linking to that page)

Is it a proper place to edit the doc ?
https://github.com/akka/akka-grpc/blob/master/docs/src/main/paradox/buildtools/gradle.md

[raboof]

yes, the "Loading proto files from artifacts" section on that page seems like a good place

[eshepelyuk]

yes, the "Loading proto files from artifacts" section on that page seems like a good place

Could you split this issue, so I'll provide a PR only for Gradle related part.

[raboof]

Could you split this issue, so I'll provide a PR only for Gradle related part.

Thanks! I'm afraid github doesn't allow 'neatly' splitting issues, but let's keep this issue for maven and continue the gradle part in #1057

[morapet]

Continuing @pradasouvanlasy thought, I came to the following fully working configuration:

<build>
    <plugins>
    	<!--  unpack proto files from external dependencies   -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <executions>
                <execution>
                    <id>unpack</id>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>unpack</goal>
                    </goals>
                    <configuration>
                        <artifactItems>
                            <artifactItem>
                                <groupId>com.some.company</groupId>
                                <artifactId>company-protos</artifactId>
                                <version>${company-protos.version}</version>
                                <type>jar</type>
                                <overWrite>true</overWrite>
                                <outputDirectory>${project.build.directory}/proto</outputDirectory>
                                <includes>**/*.proto</includes>
                                <excludes>**/*.java</excludes>
                            </artifactItem>

                            <!-- to use protobuf common types -->
                            <artifactItem>
                                <groupId>com.google.protobuf</groupId>
                                <artifactId>protobuf-java</artifactId>
                                <version>${protobuf-java.version}</version>
                                <type>jar</type>
                                <overWrite>true</overWrite>
                                <outputDirectory>${project.build.directory}/proto</outputDirectory>
                                <includes>**/*.proto</includes>
                            </artifactItem>

                            <!-- to use protobuf common types -->
                            <artifactItem>
                                <groupId>com.google.api.grpc</groupId>
                                <artifactId>proto-google-common-protos</artifactId>
                                <version>${proto-google-common-protos.version}</version>
                                <type>jar</type>
                                <overWrite>true</overWrite>
                                <outputDirectory>${project.build.directory}/proto</outputDirectory>
                                <includes>**/*.proto</includes>
                            </artifactItem>
                        </artifactItems>
                        <overWriteReleases>false</overWriteReleases>
                        <overWriteSnapshots>true</overWriteSnapshots>
                    </configuration>
                </execution>
            </executions>
        </plugin>

        <plugin>
            <groupId>com.lightbend.akka.grpc</groupId>
            <artifactId>akka-grpc-maven-plugin</artifactId>
            <version>${akka-grpc-maven-plugin.version}</version>
            <configuration>
                <language>Scala</language>
                <generateClient>true</generateClient>
                <generateServer>true</generateServer>
                <protoPaths>
                    <protoPath>target/proto</protoPath>
                </protoPaths>
            </configuration>
            <executions>
                <execution>
                    <id>generate</id>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>generate</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

        <!-- makes akka-grpc-maven-plugin generated classes available as sources -->
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>build-helper-maven-plugin</artifactId>
            <version>${build-helper-maven-plugin}</version>
            <executions>
                <execution>
                    <id>add-source</id>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>add-source</goal>
                    </goals>
                    <configuration>
                        <sources>
                            <source>${project.build.directory}/generated-sources/akka-grpc-scala/</source>
                        </sources>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

This works nicely, additionally It would be nice, if we could extract dependencies into different directory(-ies) and then pass additional includes to the akka grpc plugin with include path only (currently I only see only one include path and that is one where proto file is located). The difference is that akka-grpc generates all artifacts for all dependencies, even if they are not needed( e.g one API uses another API elements and the plugin generates also Servers, extract exclude is for me not an option). This is not really bad, but if I execute another protoc plugin (e.g python) in the same directory it generates lot of not used artifacts, servers and messages for models only included.