Skip to content

Commit

Permalink
Refer to example spider
Browse files Browse the repository at this point in the history
  • Loading branch information
wvengen committed Jan 25, 2024
1 parent e14b50c commit 90a103c
Show file tree
Hide file tree
Showing 4 changed files with 8 additions and 46 deletions.
45 changes: 3 additions & 42 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,48 +76,9 @@ TODO finish this section
* `org.scrapy.project` - the project name
* `org.scrapy.spiders` - the spiders (those returned by `scrapy list`, comma-separated)

For example, you could use this in a Github Action:

```yaml
# ...
jobs:
container:
runs-on: ubuntu-latest
steps:
- name: Docker meta
id: meta
uses: docker/metadata-action@v4
with: # ...

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1

- name: Build
id: build
uses: docker/build-push-action@v2
with:
push: false
load: true
tags: spider:latest

- name: Get spiders
if: ${{ github.event_name != 'pull_request' }}
id: spiders
run: |
SPIDERS=`docker run --rm spider:latest scrapy list | tr '\n' ',' | sed 's/,$//'`
echo "spiders=$SPIDERS" >> "$GITHUB_OUTPUT"
- name: Rebuild and push
if: ${{ github.event_name != 'pull_request' }}
uses: docker/build-push-action@v2
with:
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: |
${{ steps.meta.outputs.labels }}
org.scrapy.project=CHANGEME
org.scrapy.spiders=${{ steps.spiders.outputs.spiders }}
```
An example spider is available at [q-m/scrapyd-k8s-example-spider](https://github.com/q-m/scrapyd-k8s-spider-example),
including a [Github Action](https://github.com/q-m/scrapyd-k8s-spider-example/blob/main/.github/workflows/container.yml) for building a container.


## API

Expand Down
5 changes: 3 additions & 2 deletions kubernetes.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -74,10 +74,11 @@ data:
namespace = scrapyd
# adapt the spider config to your use-case
# This is an example spider that should work out of the box.
# Adapt the spider config to your use-case, with an otherwise unused secret.
[project.example]
env_secret = spider-example-env
repository = ghcr.io/example-org/example-spider
repository = ghcr.io/q-m/scrapyd-k8s-spider-example
---
apiVersion: v1
kind: Secret
Expand Down
2 changes: 1 addition & 1 deletion scrapyd_k8s.sample-docker.conf
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,4 @@ launcher = scrapyd_k8s.launcher.Docker
# For each project, define a project section.
# This contains a repository, with the container label to use.
[project.example]
repository = example-spider
repository = ghcr.io/q-m/scrapyd-k8s-spider-example
2 changes: 1 addition & 1 deletion scrapyd_k8s.sample-k8s.conf
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,4 @@ pull_secret = ghcr-registry
[project.example]
env_secret = example-env-secret
env_config = example-env-configmap
repository = ghcr.io/example-org/example-spider
repository = ghcr.io/q-m/scrapyd-k8s-spider-example

0 comments on commit 90a103c

Please sign in to comment.