Pipeline with service doesn't work (K8s)

[zc-devs]

Component

server, agent

Describe the bug

When running pipeline with service, it going to start, but there is no pods and even attempts to create them.
There are no errors and WP pods (Server, Agent) restarts.

Pipeline:

skip_clone: true
services:
  server:
    image: digitalocean/doks-debug
    commands:
      - echo 'This is test server' | nc -l -6 1234
    ports:
      - 1234
steps:
  client:
    image: digitalocean/doks-debug
    commands:
      - nc -v -6 -w 10 server 1234

Result:

System Info

`next-a5fa810bcc`

Additional context

woodpecker-server.log
woodpecker-agent.log

https://woodpecker.test.smthd.com/repos/1/pipeline/85

Validations

• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Checked that the bug isn't fixed in the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]
• Check that this is a concrete bug. For Q&A join our Discord Chat Server or the Matrix room.

[eliasscosta]

Hi, I am not able to use rwx volumes in my clusters, when I try with NFS services, the pipelines go to very slow to finish.

In my situation, the services start-up with a volume and lock it. It is blocking the other pod steps to continue. My suggestion is to create the pod service with an empty dir, without the volume used in the other pods steps. It's make the pods services isolated from the pv mount lock

[zc-devs]

I don't think, that volume is the cause of this issue.

On next-a5fa810bcc it worked without service. I've just tested on pull_3216 - the same.
Volume is used not only in services, but also in pipelines without them - workspace with cloned repository.
I use local storage in K3s.

Perhaps, #3141 is related.

[eliasscosta]

I don't think, that volume is the cause of this issue.

On next-a5fa810bcc it worked without service. I've just tested on pull_3216 - the same. Volume is used not only in services, but also in pipelines without them - workspace with cloned repository. I use local storage in K3s.

Perhaps, #3141 is related.

Thanks, I'm running 2.1.1. I will try replicate my issue with next version. In my issue the service works when all the steps pods run in the same kubernetes node. If some steps run in another node I got error to attach the pvc.

Warning  FailedAttachVolume  2m21s  attachdetach-controller  Multi-Attach error for volume "pvc-e7756cc7-a0ce-4c3f-909a-f92fb9051147" Volume is already used by pod(s) wp-01hmhhfewr9cwqsj2q4v440rwa-0-step-0
  Warning  FailedMount         19s    kubelet                  Unable to attach or mount volumes: unmounted volumes=[wp-01hmhhfewr9cwqsj2q4v440rwa-0-default], unattached volumes=[aws-iam-token wp-01hmhhfewr9cwqsj2q4v440rwa-0-default kube-api-access-lwm48]: timed out waiting for the condition 

[zc-devs]

It was broken twice: in #3141 and #3135.

1-woodpecker-agent-549d095f.log works.
Fix of #3141.
2-woodpecker-agent-31614d0e-patched.log works.
3-woodpecker-agent-c64340cf-patched.log works.
4-woodpecker-agent-cd59a852-patched.log works.
5-woodpecker-agent-aab2f0e6-patched.log works.

6-woodpecker-agent-d1fe86b7-patched.log broken again. The same on b82790d5.
Pipeline 114

skip_clone: true
services:
  server:
    image: digitalocean/doks-debug
    commands:
      - echo 'This is test server' | nc -l -6 1234
    ports:
      - 1234
steps:
  client:
    image: digitalocean/doks-debug
    commands:
      - nc -v -6 -w 10 server 1234

Step (client) never runs. There is no logs from service (server).

Creating pod: wp-01hmmewgwqjm0p575erpfr0xej
Tail logs of pod: server

Pipeline 120
The same if use short running command in service:

skip_clone: true
services:
  server:
    image: alpine
    commands:
      - echo Hello from service
    ports:
      - 1234
steps:
  client:
    image: alpine
    commands:
      - echo Hello from step

[6543]

to not break it again - can we mock the kubernetis api somehow and create a integration test for the backend (with a smal pipeline as run example (1x clone 1x service 3xstep (2xparalel and one waiting))?