Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide docker-web project for gradle, restructure web static content #1694

Merged
merged 1 commit into from
Dec 15, 2021
Merged

Provide docker-web project for gradle, restructure web static content #1694

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ Set<Project> modsAreBasic = subprojects.findAll {it.name in [
'deephaven2-wheel',
'envoy',
'grpc-proxy',
'web',
'web-client-ui',
'pyclient',
'sphinx',
Expand Down Expand Up @@ -480,7 +481,7 @@ tasks.register('prepareCompose') {
it.description 'A lifecycle task that prepares prerequisites for local docker-compose builds'
it.dependsOn project(':docker-server-slim').tasks.findByName('buildDocker'),
project(':docker-server').tasks.findByName('buildDocker-server'),
project(':web-client-ide').tasks.findByName('buildDocker'),
project(':docker-web').tasks.findByName('buildDocker'),
project(':envoy').tasks.findByName('buildDocker'),
project(':grpc-proxy').tasks.findByName('buildDocker')
}
Expand All @@ -496,7 +497,7 @@ tasks.register('smoke') {
it.dependsOn project(':grpc-api').tasks.findByName(LifecycleBasePlugin.CHECK_TASK_NAME)
it.dependsOn project(':docker-server-slim').tasks.findByName('prepareDocker')
it.dependsOn project(':docker-server').tasks.findByName('prepareDocker')
it.dependsOn project(':web-client-ide').tasks.findByName('prepareDocker')
it.dependsOn project(':web').tasks.findByName(LifecycleBasePlugin.ASSEMBLE_TASK_NAME)
it.dependsOn project(':Generators').tasks.findByName(LifecycleBasePlugin.CHECK_TASK_NAME)
}

Expand Down
34 changes: 0 additions & 34 deletions buildSrc/src/main/groovy/GwtTools.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,7 @@ import groovy.transform.CompileStatic
import org.gradle.api.Project
import org.gradle.api.artifacts.ProjectDependency
import org.gradle.api.file.ConfigurableFileCollection
import org.gradle.api.file.CopySpec
import org.gradle.api.plugins.JavaPlugin
import org.gradle.api.tasks.TaskOutputs
import org.gradle.api.tasks.bundling.Jar
import org.gradle.api.tasks.compile.JavaCompile

/**
Expand Down Expand Up @@ -72,37 +69,6 @@ class GwtTools {
gwtc.logger.quiet('Running in gwt dev mode; saving source to {}/dh/src', extras)
}

// Add a jar task to create an artifact containing our compiled application
p.tasks.register "jsJar", Jar, {
Jar j ->
j.group = '~Deephaven'
j.description = description
// make a task dependency, AND make sure we rebuild this jar if the gwtc task output files change.
j.inputs.files gwtc.outputs.files
j.from("$gwt.compile.war/dhapi") {
CopySpec c->
c.exclude '**/extra' // don't put our source files in exported jar, thx
c.into 'dhapi'
}
// let's also pull in the ide client's build output
TaskOutputs uiOutputs = p.project(':web-client-ui').tasks.getByName('ideClient').outputs
j.from(uiOutputs.files) {
CopySpec c ->
c.into 'dhide'
}

TaskOutputs internalOpenapiOutputs = p.project(':proto:raw-js-openapi').tasks.getByName('webpackSources').outputs
j.from(internalOpenapiOutputs.files) {
CopySpec c ->
c.into 'dhapi'
}

j.classifier = 'js'
j.doFirst {
j.logger.info "Merging $gwt.compile.war and $uiOutputs into $j.archivePath"
}
}

p.tasks.findByName('gwtCheck')?.enabled = false
}

Expand Down
37 changes: 37 additions & 0 deletions docker/web/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import io.deephaven.tools.License

plugins {
id 'com.bmuschko.docker-remote-api'
}

configurations {
js
}
dependencies {
js project(path: ':web', targetConfiguration: 'js')
}

evaluationDependsOn Docker.registryProject('nginx-base')

def dockerLicenses = License.createFrom(project).syncDockerLicense()

def syncDir = layout.buildDirectory.dir('docker')
def prepareDocker = project.tasks.register('prepareDocker', Sync) {
from layout.projectDirectory.dir('src/main/docker')
from(configurations.js) {
into 'static'
}
from (dockerLicenses.get().outputs) {
into 'licenses'
}
into syncDir
}

Docker.registerDockerImage(project, 'buildDocker') {
inputDir.set syncDir
inputs.files Docker.registryFiles(project, 'nginx-base')
inputs.files prepareDocker.get().outputs.files
images.add('deephaven/web:local-build')
}

assemble.dependsOn buildDocker
10 changes: 10 additions & 0 deletions docker/web/src/main/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
FROM deephaven/nginx-base:local-build

COPY licenses/ /
COPY nginx/default.conf /etc/nginx/conf.d/
COPY nginx/nginx.conf /etc/nginx/
COPY nginx/99-init-notebooks.sh /docker-entrypoint.d

COPY static/ /usr/share/nginx/html

VOLUME /tmp
0 web/client-ide/nginx/99-init-notebooks.sh → ...rc/main/docker/nginx/99-init-notebooks.sh
View file
Open in desktop
File renamed without changes.
0 web/client-ide/nginx/default.conf → ...er/web/src/main/docker/nginx/default.conf
View file
Open in desktop
File renamed without changes.
0 web/client-ide/nginx/nginx.conf → docker/web/src/main/docker/nginx/nginx.conf
View file
Open in desktop
File renamed without changes.
16 changes: 13 additions & 3 deletions proto/raw-js-openapi/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ evaluationDependsOn ':proto:proto-backplane-grpc'
evaluationDependsOn Docker.registryProject('node')

def backplaneProject = project(':proto:proto-backplane-grpc')
def webpackSourcesLocation = "${buildDir}/dhapi"
def webpackSourcesLocation = layout.buildDirectory.dir("${buildDir}/dhapi")

Docker.registerDockerTask(project, 'webpackSources') {
copyIn {
Expand All @@ -30,12 +30,22 @@ Docker.registerDockerTask(project, 'webpackSources') {
from ('Dockerfile')
from ('../package.json')
from ('../package-lock.json')

}
parentContainers = [ Docker.registryTask(project, 'node') ]
imageName = 'deephaven/js-out:local-build'
containerOutPath = '/usr/src/app/raw-js-openapi/build/js-out'
copyOut {
into(webpackSourcesLocation)
include 'dh-internal.js'
into(webpackSourcesLocation.get().dir('jsapi'))
}
}

configurations {
js
}

artifacts {
js(webpackSourcesLocation) {
builtBy webpackSources
devinrsmith marked this conversation as resolved.
Show resolved Hide resolved
}
}
4 changes: 4 additions & 0 deletions settings.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ String[] mods = [ 'Util', 'Numerics', 'TableLogger', 'Plot',
// new web projects; these modules are intended to form a complete, modular web application,
// with heavy dependency isolation that should enable very fast rebuilds.
String[] webMods = [
'web',
'web-client-api', // compiled javascript api client
'web-client-ide', // IDE-only additions to open-api javascript client
'web-client-ui', // React IDE client app
Expand Down Expand Up @@ -245,6 +246,9 @@ project(':docker-runtime-base').projectDir = file('docker/runtime-base')
include(':docker-server')
project(':docker-server').projectDir = file('docker/server')

include(':docker-web')
project(':docker-web').projectDir = file('docker/web')

include(':docker-server-slim')
project(':docker-server-slim').projectDir = file('docker/server-slim')

Expand Down
56 changes: 18 additions & 38 deletions web/client-ide/client-ide.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,57 +1,37 @@
import io.deephaven.tools.License

plugins {
id 'com.bmuschko.docker-remote-api'
}

evaluationDependsOn ':web-client-ui'
evaluationDependsOn ':proto:raw-js-openapi'
evaluationDependsOn Docker.registryProject('nginx-base')

apply from: "$rootDir/gradle/web-client.gradle"

configurations {
js
}

dependencies {
compile project(':web-client-api')
compile project(':open-api-lang-parser')

js project(path: ':proto:raw-js-openapi', configuration: 'js')
}

GwtTools.gwtCompile project, 'io.deephaven.ide.DeephavenIde', 'Create a jar of javascript for web IDE'

// deephaven/dhide
Task dhide = project(':web-client-ui').tasks.findByName('ideClientMakeImage')

// deephaven/js-out
Task jsOut = project(':proto:raw-js-openapi').tasks.findByName('webpackSourcesMakeImage')

Task gwtc = tasks.getByName('gwtCompile')

def dockerLicenses = License.createFrom(project).syncDockerLicense()

def prepareDocker = project.tasks.register('prepareDocker', Sync) {
// TODO(deephaven-core#1596): Remove extra dependencies for build-ci.yml
// Note: GH actions currently prepares docker via gradle, but does its own building via
// a native GH action. As such, we need to make sure that the prepare also invokes the
// necessarily build pre-reqs. If GH actions eventually calls buildDocker instead, we
// can remove these dependencies.
inputs.files([dhide, jsOut].each { t -> t.outputs.files })
inputs.files Docker.registryFiles(project, 'nginx-base')

into "${buildDir}/docker"
from(gwtc.outputs.files) {
include("dhapi/**")
def jsOutput = layout.buildDirectory.dir('js')
def gwtOutput = tasks.register('gwtOutput', Sync) {
includeEmptyDirs = false
from(tasks.getByName('gwtCompile').outputs.files) {
// only copy the dhapi module, and don't give it a wrapper directory
include 'dhapi/**'
eachFile { it.path = 'jsapi/' + it.path.substring('dhapi/'.length()) }
}
from (dockerLicenses.get().outputs) {
into 'licenses'
}
from 'docker/Dockerfile'
from 'nginx/default.conf'
from 'nginx/nginx.conf'
from 'nginx/99-init-notebooks.sh'
into jsOutput
}

Docker.registerDockerImage(project, 'buildDocker') {
inputs.files([prepareDocker.get(), dhide, jsOut].each { t -> t.outputs.files })
inputs.files Docker.registryFiles(project, 'nginx-base')
//buildArgs.put('DEEPHAVEN_VERSION', "${project.version}")
images.add('deephaven/web:local-build')
artifacts {
js(jsOutput) {
builtBy gwtOutput
}
}
15 changes: 0 additions & 15 deletions web/client-ide/docker/Dockerfile
View file
Open in desktop

This file was deleted.

15 changes: 13 additions & 2 deletions web/client-ui/client-ui.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,25 @@ plugins {

evaluationDependsOn Docker.registryProject('node')

Docker.registerDockerTask(project, 'ideClient') {
def dhUi = layout.buildDirectory.dir('dhide')
def ui = Docker.registerDockerTask(project, 'ui') {
copyIn {
from file('Dockerfile')
}
parentContainers = [ Docker.registryTask(project, 'node') ]
containerOutPath = '/usr/src/app/package/build'
imageName = 'deephaven/dhide:local-build'
copyOut {
into "${buildDir}/dhide"
into dhUi.get().dir('ide')
}
}

configurations {
js
}

artifacts {
js(dhUi) {
builtBy ui
niloc132 marked this conversation as resolved.
Show resolved Hide resolved
}
}
25 changes: 25 additions & 0 deletions web/web.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
configurations {
js
jsJar {
transitive = false
}
}

archivesBaseName = "deephaven-web"

dependencies {
js project(path: ':web-client-ui', configuration: 'js')
js project(path: ':web-client-ide', configuration: 'js')
}

/**
* Provides a jar full of static js/html/css that can be served from jetty
*/
def ideClientJsJar = tasks.register('ideClientJsJar', Jar) {
from configurations.js
}
Comment on lines +18 to +20
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The jar name is different than before. deephaven-web-client-ide-0.8.0-js.jar vs web-0.8.0.jar. Since it was a under the guise of a java project before, it inherited buildSrc/src/main/groovy/io.deephaven.java-conventions.gradle which includes setting archivesBaseName. The easy fix is to set

archivesBaseName = "deephaven-web"

. Long term, we should probably have base-conventions.gradle separated out.

Do we need/want the -js classifier on the jar name?

Upon my first build of web:ideClientJsJar, I was seeing extra content. It may have been old gradle directories that were being passively picked up. Was a bit strange. After a clean it went away.

Here are the jar tf for the builds:

$ ls -la /tmp/*.txt
-rw-r--r--. 1 devin devin 4943 Dec 14 13:50 /tmp/deephaven-web-client-ide-0.8.0-js.txt
-rw-r--r--. 1 devin devin 4667 Dec 14 13:56 /tmp/web2.txt
-rw-r--r--. 1 devin devin 8336 Dec 14 13:51 /tmp/web.txt

You can see the first web build is about twice as big as web2.

And as far as content, I see that there are some different prefixes now.

For example:

$ grep wordmark /tmp/*.txt
/tmp/deephaven-web-client-ide-0.8.0-js.txt:dhide/static/media/community-wordmark-app.1acb5a93.svg
/tmp/web2.txt:ide/static/media/community-wordmark-app.1acb5a93.svg

$ grep treegrid /tmp/*.txt
/tmp/deephaven-web-client-ide-0.8.0-js.txt:dhapi/treegrid.js
/tmp/web2.txt:jsapi/treegrid.js

dhapi -> jsapi and dhide -> ide, that's correct?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The jar name is different

Do we care? I'm not sure either way just yet. This jar is being created only so that it can be on the classpath of the eventual jetty-based executable, so that content can be served off of its classpath.

If we publish the web ui's static content as a jar to central I'm sure we care. Otherwise I'm uncertain, will be guided by you.

Do we want/need the -js classifier

No, this project does not produce java that is useful downstream to anyone (at this time?)

--

Probably your uncleaned version had leftover UI stuff copied into it - I've seen that a lot recently where gradle is over-confident in what is correctly already built and what isn't. 99% the "fault" is actually mine, but it is very hard to reliably use when it is so hard to hold the tool correctly. I'll take another pass at checking the before/after of this.

--

dhapi -> jsapi and dhide -> ide, that's correct?

"jsapi" and "ide" are the desired paths that the user sees in the browser. We can probably rewrite the "dhapi" and "dhide" internally, but these path changes are consistent with the build before my patch. If you'd like, I'll address that now, your call.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do care about about the jar name prefix - deephaven-<x>, I don't care about the classifier being appended.

Please set the archiveBaseName to deephaven-web (or something similar if you think it makes more sense), but put a reference in the code to #1701


artifacts {
jsJar ideClientJsJar
js layout.buildDirectory.dir('empty')
}