feat: have all pvc-related stuff under /data

Signed-off-by: Hannah Zhang <hannahz@nvidia.com>

Author: hhzhang16

benchmarks/profiler/deploy/profile_sla_job.yaml

@@ -29,9 +29,9 @@ spec:
         command: ["python", "-m", "benchmarks.profiler.profile_sla"]
         args:
           - --config
-          - /workspace/configs/disagg.yaml
+          - /workspace/data/configs/disagg.yaml
           - --output-dir
-          - /workspace/profiling_results
+          - /workspace/data/profiling_results
           - --namespace
           - ${NAMESPACE}
           - --backend
@@ -50,15 +50,10 @@ spec:
           - "20"
         volumeMounts:
           - name: output-volume
-            mountPath: /workspace/profiling_results
-          - name: configs
-            mountPath: /workspace/configs
+            mountPath: /workspace/data
       restartPolicy: Never
       volumes:
         - name: output-volume
           persistentVolumeClaim:
             claimName: dynamo-pvc
-        - name: configs
-          persistentVolumeClaim:
-            claimName: dynamo-pvc
   backoffLimit: 0

deploy/utils/README.md

@@ -88,23 +88,36 @@ These scripts interact with the Persistent Volume Claim (PVC) that stores config
 
 ```bash
 # The profiling job reads your DGD config from the PVC
-python3 deploy/utils/inject_manifest.py \
+# IMPORTANT: All paths must start with /data/ for security reasons
+python3 -m deploy.utils.inject_manifest \
   --namespace $NAMESPACE \
   --src ./my-disagg.yaml \
-  --dest /configs/disagg.yaml
+  --dest /data/configs/disagg.yaml
 ```
 
 **Download benchmark/profiling results:**
 
 ```bash
 # After benchmarking or profiling completes, download results
-python3 deploy/utils/download_pvc_results.py \
+python3 -m deploy.utils.download_pvc_results \
   --namespace $NAMESPACE \
   --output-dir ./pvc_files \
-  --folder /results \
+  --folder /data/results \
   --no-config   # optional: skip *.yaml/*.yml in the download
 ```
 
+#### Path Requirements
+
+**Important**: The PVC is mounted at `/data` in the access pod for security reasons. All destination paths must start with `/data/`.
+
+**Common path patterns:**
+- `/data/configs/` - Configuration files (DGD manifests)
+- `/data/results/` - Benchmark results
+- `/data/profiling_results/` - Profiling data
+- `/data/benchmarking/` - Benchmarking artifacts
+
+**User-friendly error messages**: If you forget the `/data/` prefix, the script will show a helpful error message with the correct path and example commands.
+
 #### Next Steps
 
 For complete benchmarking workflows:

deploy/utils/download_pvc_results.py

@@ -36,7 +36,7 @@
     from deploy.utils.kubernetes import (
         check_kubectl_access,
         cleanup_access_pod,
-        deploy_access_pod,
+        ensure_clean_access_pod,
         run_command,
     )
 except ModuleNotFoundError:
@@ -46,7 +46,7 @@
     from deploy.utils.kubernetes import (
         check_kubectl_access,
         cleanup_access_pod,
-        deploy_access_pod,
+        ensure_clean_access_pod,
         run_command,
     )
 
@@ -194,7 +194,7 @@ def main():
     check_kubectl_access(args.namespace)
 
     # Deploy access pod
-    pod_name = deploy_access_pod(args.namespace)
+    pod_name = ensure_clean_access_pod(args.namespace)
     try:
         # List and download files
         files = list_pvc_contents(args.namespace, pod_name, args.folder, args.no_config)

deploy/utils/inject_manifest.py

@@ -21,12 +21,15 @@
 Copies any Kubernetes manifest file into the PVC for later use by jobs.
 Both the source manifest path and destination path in the PVC are required.
 
+IMPORTANT: The PVC is mounted at /data in the access pod for security reasons.
+All destination paths must start with '/data/'.
+
 Usage:
     python3 inject_manifest.py --namespace <namespace> --src <local_manifest.yaml> --dest <absolute_path_in_pvc>
 
 Examples:
-    python3 inject_manifest.py --namespace <ns> --src ./my-disagg.yaml --dest /configs/disagg.yaml
-    python3 inject_manifest.py --namespace <ns> --src ./my-agg.yaml    --dest /configs/agg.yaml
+    python3 inject_manifest.py --namespace <ns> --src ./disagg.yaml --dest /data/configs/disagg.yaml
+    python3 inject_manifest.py --namespace <ns> --src ./my-data.yaml    --dest /data/custom/path/data.yaml
 """
 
 import argparse
@@ -37,7 +40,7 @@
     PVC_ACCESS_POD_NAME,
     check_kubectl_access,
     cleanup_access_pod,
-    deploy_access_pod,
+    ensure_clean_access_pod,
     run_command,
 )
 
@@ -78,6 +81,7 @@ def copy_manifest(namespace: str, manifest_path: Path, target_path: str) -> None
 
     print("✓ Manifest successfully copied to PVC")
     print(f"File details: {result.stdout.strip()}")
+    return target_path
 
 
 def main():
@@ -100,16 +104,39 @@ def main():
     parser.add_argument(
         "--dest",
         required=True,
-        help="Absolute target path in PVC (e.g., /profiling_results/agg.yaml)",
+        help="Absolute target path in PVC (must start with /data/, e.g., /data/configs/agg.yaml)",
     )
 
     args = parser.parse_args()
 
-    # Validate target_path to prevent directory traversal
-    if not args.dest.startswith("/"):
-        print(
-            "ERROR: Target path must be an absolute path inside the PVC (start with '/')."
-        )
+    # Validate target_path to prevent directory traversal and ensure it's within PVC
+    if not args.dest.startswith("/data/"):
+        print("=" * 60)
+        print("❌ ERROR: Invalid target path")
+        print("=" * 60)
+        print("The PVC is mounted at /data in the access pod.")
+        print("All paths must start with '/data/' for security reasons.")
+        print("")
+        print("💡 QUICK FIX:")
+        if args.dest.startswith("/"):
+            # Suggest the fix
+            suggested_path = f"/data{args.dest}"
+            print(f"  Change: {args.dest}")
+            print(f"  To:     {suggested_path}")
+            print("")
+            print("📝 Example commands:")
+            print("  python3 -m deploy.utils.inject_manifest \\")
+            print(f"    --namespace {args.namespace} \\")
+            print(f"    --src {args.src} \\")
+            print(f"    --dest {suggested_path}")
+        else:
+            print(f"  Use: /data/{args.dest.lstrip('/')}")
+        print("")
+        print("🔍 Common patterns:")
+        print("  /configs/file.yaml     → /data/configs/file.yaml")
+        print("  /results/data.yaml     → /data/results/data.yaml")
+        print("  /profiling_results/... → /data/profiling_results/...")
+        print("=" * 60)
         sys.exit(1)
 
     if ".." in args.dest:
@@ -123,7 +150,7 @@ def main():
     check_kubectl_access(args.namespace)
 
     # Deploy access pod
-    deploy_access_pod(args.namespace)
+    ensure_clean_access_pod(args.namespace)
     try:
         # Copy manifest
         copy_manifest(args.namespace, args.src, args.dest)

deploy/utils/kubernetes.py

@@ -22,7 +22,7 @@
 
 
 def run_command(
-    cmd: List[str], capture_output: bool = True
+    cmd: List[str], capture_output: bool = True, exit_on_error: bool = True
 ) -> subprocess.CompletedProcess:
     """Run a command and handle errors."""
     try:
@@ -37,7 +37,10 @@ def run_command(
             print(f"STDOUT: {e.stdout}")
         if e.stderr:
             print(f"STDERR: {e.stderr}")
-        sys.exit(1)
+        if exit_on_error:
+            sys.exit(1)
+        else:
+            raise
 
 
 def check_kubectl_access(namespace: str) -> None:
@@ -47,6 +50,55 @@ def check_kubectl_access(namespace: str) -> None:
     print("✓ kubectl access confirmed")
 
 
+def ensure_clean_access_pod(namespace: str) -> str:
+    """Ensure a clean PVC access pod deployment by deleting any existing pod first."""
+
+    # Check if pod exists and delete it if it does
+    try:
+        result = subprocess.run(
+            [
+                "kubectl",
+                "get",
+                "pod",
+                PVC_ACCESS_POD_NAME,
+                "-n",
+                namespace,
+                "-o",
+                "jsonpath={.metadata.name}",
+            ],
+            capture_output=True,
+            text=True,
+            check=False,
+        )
+        if result.returncode == 0 and result.stdout.strip() == PVC_ACCESS_POD_NAME:
+            print(f"Found existing access pod '{PVC_ACCESS_POD_NAME}', deleting it...")
+            run_command(
+                [
+                    "kubectl",
+                    "delete",
+                    "pod",
+                    PVC_ACCESS_POD_NAME,
+                    "-n",
+                    namespace,
+                    "--ignore-not-found",
+                ],
+                capture_output=False,
+                exit_on_error=False,
+            )
+            print("✓ Existing access pod deleted")
+    except Exception:
+        pass  # Pod doesn't exist, which is fine
+
+    try:
+        return deploy_access_pod(namespace)
+    except Exception as e:
+        print(f"Deployment failed: {e}")
+        print(
+            "Pod left running for debugging. Use 'kubectl delete pod pvc-access-pod -n <namespace>' to clean up manually."
+        )
+        raise
+
+
 def deploy_access_pod(namespace: str) -> str:
     """Deploy the PVC access pod and return pod name."""
 
@@ -67,25 +119,19 @@ def deploy_access_pod(namespace: str) -> str:
             text=True,
             check=False,
         )
-
         if result.returncode == 0 and result.stdout.strip() == "Running":
             print(f"✓ Access pod '{PVC_ACCESS_POD_NAME}' already running")
             return PVC_ACCESS_POD_NAME
     except Exception:
-        # Pod doesn't exist or isn't running
-        pass
+        pass  # Pod doesn't exist or isn't running
 
     print(f"Deploying access pod '{PVC_ACCESS_POD_NAME}' in namespace '{namespace}'...")
 
-    # Get the directory where this script is located
-    script_dir = Path(__file__).parent
-    pod_yaml_path = script_dir / "manifests" / "pvc-access-pod.yaml"
-
+    pod_yaml_path = Path(__file__).parent / "manifests" / "pvc-access-pod.yaml"
     if not pod_yaml_path.exists():
         print(f"ERROR: Pod YAML not found at {pod_yaml_path}")
         sys.exit(1)
 
-    # Deploy the pod
     run_command(
         ["kubectl", "apply", "-f", str(pod_yaml_path), "-n", namespace],
         capture_output=False,
@@ -103,23 +149,28 @@ def deploy_access_pod(namespace: str) -> str:
             "--timeout=60s",
         ],
         capture_output=False,
+        exit_on_error=False,
     )
     print("✓ Access pod is ready")
     return PVC_ACCESS_POD_NAME
 
 
 def cleanup_access_pod(namespace: str) -> None:
     print("Cleaning up access pod...")
-    run_command(
-        [
-            "kubectl",
-            "delete",
-            "pod",
-            PVC_ACCESS_POD_NAME,
-            "-n",
-            namespace,
-            "--ignore-not-found",
-        ],
-        capture_output=False,
-    )
-    print("✓ Access pod deleted")
+    try:
+        run_command(
+            [
+                "kubectl",
+                "delete",
+                "pod",
+                PVC_ACCESS_POD_NAME,
+                "-n",
+                namespace,
+                "--ignore-not-found",
+            ],
+            capture_output=False,
+            exit_on_error=False,
+        )
+        print("✓ Access pod deleted")
+    except Exception as e:
+        print(f"Warning: Failed to clean up access pod: {e}")

deploy/utils/manifests/pvc-access-pod.yaml

@@ -26,7 +26,7 @@ spec:
         - ALL
     volumeMounts:
     - name: profiling-storage
-      mountPath: /profiling_results
+      mountPath: /data
     resources:
       requests:
         memory: "128Mi"

docs/benchmarks/pre_deployment_profiling.md

@@ -90,7 +90,7 @@ Use the injector utility to place your DGD manifest into the PVC. The profiling
 python3 deploy/utils/inject_manifest.py \
   --namespace $NAMESPACE \
   --src components/backends/vllm/deploy/disagg.yaml \
-  --dest /configs/disagg.yaml
+  --dest /data/configs/disagg.yaml
 
 # Set the docker image for the profiling job; any docker image that contains your script.
 export DOCKER_IMAGE=nvcr.io/nvidia/dynamo:latest-vllm
@@ -112,15 +112,17 @@ Use the default pre-built image and inject custom configurations via PVC:
 2. **Inject your custom disagg configuration:**
    ```bash
    # Use default disagg.yaml config
-   python3 deploy/utils/inject_manifest.py --namespace $NAMESPACE --src components/backends/vllm/deploy/disagg.yaml --dest /configs/disagg.yaml
+   python3 deploy/utils/inject_manifest.py --namespace $NAMESPACE --src components/backends/vllm/deploy/disagg.yaml --dest /data/configs/disagg.yaml
 
    # Or use a custom disagg config file
-   python3 deploy/utils/inject_manifest.py --namespace $NAMESPACE --src my-custom-disagg.yaml --dest /configs/disagg.yaml
+   python3 deploy/utils/inject_manifest.py --namespace $NAMESPACE --src my-custom-disagg.yaml --dest /data/configs/disagg.yaml
 
    # Or specify a custom target path in the PVC
-   python3 deploy/utils/inject_manifest.py --namespace $NAMESPACE --src my-custom-disagg.yaml --dest /profiling_results/my-disagg.yaml
+   python3 deploy/utils/inject_manifest.py --namespace $NAMESPACE --src my-custom-disagg.yaml --dest /data/profiling_results/my-disagg.yaml
    ```
 
+   > **Note**: All paths must start with `/data/` for security reasons. If you forget this prefix, the script will show a helpful error message with the correct path.
+
 3. **Set the config path for the profiling job:**
    ```bash
    export DGD_CONFIG_FILE=/workspace/profiling_results/disagg.yaml # or your custom path
@@ -176,10 +178,10 @@ To download the results:
 
 ```bash
 # Download to directory
-python3 deploy/utils/download_pvc_results.py --namespace $NAMESPACE --output-dir ./results --folder /profiling_results
+python3 deploy/utils/download_pvc_results.py --namespace $NAMESPACE --output-dir ./results --folder /data/profiling_results
 
 # Download without any of the auto-created config.yaml files used in profiling
-python3 deploy/utils/download_pvc_results.py --namespace $NAMESPACE --output-dir ./results --folder /profiling_results --no-config
+python3 deploy/utils/download_pvc_results.py --namespace $NAMESPACE --output-dir ./results --folder /data/profiling_results --no-config
 ```
 
 The script will:
@@ -191,7 +193,7 @@ The script will:
 
 The profiling results directory contains the following structure:
 ```
-/workspace/profiling_results/
+/workspace/data/profiling_results/
 ├── prefill_performance.png                    # Main prefill performance plot
 ├── decode_performance.png                     # Main decode performance plot
 ├── prefill_tp1/                               # Individual TP profiling directories