Service for searching across kubernetes git repositories #96

dims · 2017-12-15T15:31:38Z

No description provided.

k8s-reviewable · 2017-12-15T15:31:46Z

This change is

dims · 2017-12-15T15:35:59Z

/assign @ixdy

Hey @ixdy, i'd like to add a new service for searching across all our git repositories. Can you please take a look when you get a chance?

Thanks,
dims

dims · 2017-12-15T15:36:28Z

oh i have one running if you want to try it - http://dims.dynu.net/hound/

thockin · 2017-12-15T16:28:07Z

hound/Dockerfile

+
+COPY config.json /data/config.json
+
+RUN apt-get update && apt-get install -y --no-install-recommends \


If you are going to build in a container, why not use the golang container?

This is going to produce a pretty big image - should it use build stages to minimize?

Ack. switching to FROM golang

thockin · 2017-12-15T16:29:25Z

hound/config.json

+	"dbpath": "data",
+	"repos": {
+		"kubernetes": {
+			"url": "https://github.com/kubernetes/kubernetes.git",


no way to say "all repos under an org" ? feature request time?

will check if they support this.

thockin · 2017-12-15T16:30:17Z

hound/deployment.yaml

+          resources:
+            limits:
+              cpu: 0.1
+              memory: 128Mi


empirical or wild guess?

copy-paste. removing

thockin · 2017-12-15T16:30:34Z

hound/deployment.yaml

+  template:
+    metadata:
+      labels:
+        app: gcsweb


copy-paster artifact?

yep. fixing

thockin · 2017-12-15T16:31:13Z

hound/deployment.yaml

+      terminationGracePeriodSeconds: 30
+      containers:
+        - name: hound
+          image: gcr.io/google_containers/hound:0.1.0


most of our images are tagged vX.Y.Z now

ack switching

thockin · 2017-12-15T16:31:59Z

hound/service.yaml

+  type: LoadBalancer
+  # FIXME(dims) - we need a fixed IP address for the load balancer
+  #loadBalancerIP: 104.197.177.166
+  ports:


we really need to get kube-lego or something running in the utility cluster, for HTTPS

indeed. #38 :)

dims · 2018-01-05T19:51:23Z

/assign @fejta

@fejta @ixdy - all i need is a loadBalancerIP and this is ready to go ... please help when you get a chance.

ixdy · 2018-01-05T19:58:17Z

hound/Makefile

+# limitations under the License.
+
+TAG = v0.1.0
+IMAGE_NAME = gcr.io/google_containers/hound


can we please not put this in gcr.io/google-containers? there's already too much there.

@ixdy what's a good spot? i was just following the previous pattern.

I'm not sure. we may need a utility project or something.

ixdy

is there a reason to build our own image rather than just using the upstream image from dockerhub and loading config.json as a configmap?

ixdy · 2018-01-05T22:06:01Z

hound/Dockerfile

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM golang


for reproducibility, maybe specify a version here?

ack. doing that

ixdy · 2018-01-05T22:09:32Z

hound/Makefile

+# limitations under the License.
+
+TAG = v0.1.0
+IMAGE_NAME = gcr.io/google_containers/hound


I'm not sure. we may need a utility project or something.

ixdy · 2018-01-05T22:09:49Z

hound/Dockerfile

+
+ENV GOPATH /go
+
+RUN go get github.com/etsy/hound/cmds/...


this is also not reproducible. :(

they dont' have any recent tags, let me pin it to a recent SHA/hash

thockin · 2018-01-05T23:23:33Z

I don't know - I don't really want to deploy something we don't control.

…

On Fri, Jan 5, 2018 at 11:58 AM, Jeff Grafton ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In hound/Makefile <#96 (comment)>: > +# Copyright 2017 The Kubernetes Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +TAG = v0.1.0 +IMAGE_NAME = gcr.io/google_containers/hound can we please not put this in gcr.io/google-containers? there's already too much there. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#96 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVGyK4VjmSxUPw-ELX1WHDGZJj3wUks5tHn7ZgaJpZM4RDoju> .

dims · 2018-01-06T02:02:28Z

published @ docker.io/dims/hound:v0.1.0

https://hub.docker.com/r/dims/hound/tags/

fejta · 2018-01-08T19:47:13Z

/unassign
/assign @thockin

ixdy · 2018-01-09T00:37:44Z

@dims did you mean to push a new commit here, too?

dims · 2018-01-09T01:00:29Z

whoops! done @ixdy

thockin · 2018-01-09T06:23:03Z

If we are going to host this, the directory should be named hound.k8s.io or codesearch.k8s,io or whatever we call it..

dims · 2018-01-09T11:23:20Z

@thockin done

dims · 2018-01-20T01:58:22Z

@thockin the rename is done. PTAL

dims · 2018-01-26T01:11:10Z

let's revive this if needed.

mtaufen · 2018-02-16T22:43:17Z

Was there a reason we quit this effort? This looks pretty cool.

thockin · 2018-04-03T18:50:10Z

I really want this. @dims - worth reviving?

dims · 2018-04-03T19:03:43Z

@thockin happy to!

dims · 2018-04-03T19:04:37Z

worst case we can throw it on GAE ( https://gist.github.com/dims/2cf8363d91c95535eecd586f57ec0ccf ) though we have to fix up the ports from 6080 to 8080 for that

dims · 2018-04-04T13:48:46Z

i am running an instance on GAE if you want to kick the tires - https://k8s-code.appspot.com - don't know how long i'll leave it up there (not sure how much it will cost)

thockin · 2018-04-06T15:09:16Z

404

…

On Wed, Apr 4, 2018 at 6:49 AM Davanum Srinivas ***@***.***> wrote: i am running an instance on GAE if you want to kick the tires - https://k8scode.appspot.com - don't know how long i'll leave it up there (not sure how much it will cost) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#96 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVJEhSi2Y9yrEf0bpEqqnUDOM-3viks5tlM8-gaJpZM4RDoju> .

dims · 2018-04-06T15:13:39Z

https://k8s-code.appspot.com (note the '-')

mtaufen · 2018-04-06T15:21:51Z

This is awesome (and really fast). Let the cross-repo auditing begin!

…

On Fri, Apr 6, 2018, 8:13 AM Davanum Srinivas ***@***.***> wrote: https://k8s-code.appspot.com (note the '-') — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#96 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA3JwUXL075hTspItQ33-DTqYqifoJ9Qks5tl4YlgaJpZM4RDoju> .

thockin · 2018-04-06T17:36:21Z

codesearch.k8s.io/deployment.yaml

+      terminationGracePeriodSeconds: 30
+      containers:
+        - name: hound
+          image: gcr.io/google_containers/hound:v0.1.0


You should probably have a readiness probe since you say it takes a while to build the index?

thockin · 2018-04-06T17:36:32Z

codesearch.k8s.io/deployment.yaml

+          image: gcr.io/google_containers/hound:v0.1.0
+          ports:
+            - containerPort: 6080
+              protocol: TCP


newline at end?

thockin · 2018-04-06T17:38:04Z

codesearch.k8s.io/service.yaml

+spec:
+  selector:
+    app: hound
+  type: LoadBalancer


Why not ingress so we can get HTTPS?

just following the previous pattern. i am not sure of what's available in the infrastructure

thockin · 2018-04-06T17:38:37Z

codesearch.k8s.io/config.json

+{
+	"max-concurrent-indexers": 2,
+	"dbpath": "data",
+	"repos": {


Question from before - can we make it pull from all repos?

also, do we need to index kubernetes-sigs and incubator?

adding a python script to fetch all the ones from kubernetes/*, once we get this working, we can pull other orgs

thockin · 2018-04-06T17:40:36Z

codesearch.k8s.io/Dockerfile

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM golang:1.9.2


does this produce an enormous result image, still?

https://hub.docker.com/r/k8scode/hound/tags/ shows compressed size is 130 MB

dims · 2018-04-10T13:36:27Z

I am just going to maintain the code here - https://github.com/dims/k8s-code.appspot.com since https://k8s-code.appspot.com/ is chugging along fine.

/close

dims · 2018-04-30T20:27:08Z

/reopen

dims · 2018-05-07T22:27:38Z

@thockin @ixdy can you please help move this to production? So i can shut down the one running on my personal GAE account? ( http://k8s-code.appspot.com/ )

dims · 2018-05-29T13:34:10Z

Let's try this again when we have access to GCE/GKE credits for running such services

/close

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 15, 2017

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 15, 2017

k8s-ci-robot assigned ixdy Dec 15, 2017

thockin reviewed Dec 15, 2017

View reviewed changes

dims force-pushed the add-hound-search-app branch from 67fc1cb to 1f5e40e Compare December 15, 2017 17:23

dims changed the title ~~[WIP] service for searching across kubernetes git repositories~~ Service for searching across kubernetes git repositories Dec 23, 2017

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 23, 2017

k8s-ci-robot assigned fejta Jan 5, 2018

ixdy reviewed Jan 5, 2018

View reviewed changes

k8s-ci-robot assigned thockin and unassigned fejta Jan 8, 2018

dims force-pushed the add-hound-search-app branch from 1f5e40e to a8894ab Compare January 9, 2018 00:59

dims closed this Jan 26, 2018

dims reopened this Apr 3, 2018

dims force-pushed the add-hound-search-app branch from 672fafb to 64e4316 Compare April 3, 2018 19:00

thockin mentioned this pull request Apr 5, 2018

Promote new search engine for kubernetes main github repos #116

Merged

thockin reviewed Apr 6, 2018

View reviewed changes

dims closed this Apr 10, 2018

k8s-ci-robot reopened this Apr 30, 2018

dims mentioned this pull request Apr 30, 2018

service for searching across kubernetes git repositories #117

Closed

dims force-pushed the add-hound-search-app branch from 64e4316 to a034408 Compare May 1, 2018 22:49

service for searching across kubernetes git repositories

a034408

k8s-ci-robot closed this May 29, 2018


		COPY config.json /data/config.json

		RUN apt-get update && apt-get install -y --no-install-recommends \

Service for searching across kubernetes git repositories #96

Service for searching across kubernetes git repositories #96

Conversation

dims commented Dec 15, 2017

k8s-reviewable commented Dec 15, 2017

dims commented Dec 15, 2017

dims commented Dec 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dims commented Jan 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ixdy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thockin commented Jan 5, 2018 via email

dims commented Jan 6, 2018 • edited Loading

fejta commented Jan 8, 2018

ixdy commented Jan 9, 2018

dims commented Jan 9, 2018

thockin commented Jan 9, 2018

dims commented Jan 9, 2018

dims commented Jan 20, 2018

dims commented Jan 26, 2018

mtaufen commented Feb 16, 2018

thockin commented Apr 3, 2018

dims commented Apr 3, 2018

dims commented Apr 3, 2018

dims commented Apr 4, 2018 • edited Loading

thockin commented Apr 6, 2018 via email

dims commented Apr 6, 2018

mtaufen commented Apr 6, 2018 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dims Apr 30, 2018 • edited Loading

Choose a reason for hiding this comment

dims commented Apr 10, 2018

dims commented Apr 30, 2018

dims commented May 7, 2018

dims commented May 29, 2018

dims commented Jan 6, 2018 •

edited

Loading

dims commented Apr 4, 2018 •

edited

Loading

dims Apr 30, 2018 •

edited

Loading