Spark Rapids is taking more scan time using GpuScan parquet than the scan on CPU Scan parquet

[pedastrian57]

This is a new thread for discussion which was already happening here as told by @jlowe and also the discussion I had over slack with @abellina

1. Decrease the executor cores (recommended 16) as there might be waiting time for tasks to have GPUs
2. Using ARENA allocator as the pooling can be very helpful(without it, you have a lot of overhead of cudaMalloc/cudaFree)
3. Removing --conf spark.rapids.memory.gpu.debug=STDOUT as it could be a costly operation
4. Using spark.rapids.sql.metrics.level=DEBUG to get all the metrics related to rapids
5. spark.rapids.sql.format.parquet.read.enabled=false instead of --conf spark.rapids.sql.input.ParquetScan=false

[abellina]

just to be clear:

spark.rapids.sql.format.parquet.read.enabled=false instead of --conf spark.rapids.sql.input.ParquetScan=false

the idea behind this is to actually disable the scan node to see if we can isolate the performance issue to the scan, but ideally we don't want to disable anything.

[pedastrian57]

We have tried reducing the number of cores from 30 to 10, and the CPU is taking more time while GPU is performing better or equal so we have improved speedup with this.

Also, we have tried increasing the concurrent GPU tasks from 1 to 5, and with that GPU semaphore wait time has reduced and so does the overall scan time.

But the above experiments which we have done are on local HDFS but when we try to access the remote HDFS the scan time becomes variable. We are assuming this is because of the network lag, we will try to increase the network bandwidth and try to continue our experiments with the remote HDFS data.

spark.rapids.sql.format.parquet.read.enabled=false this did not help us that much though. As I understood it just separates out the two operators, CPU scan and host to GPU memory transfer

[abellina]

@pedastrian57 we discussed this part of this on the side and I know you were going to try more experiments and give us more feedback. Anything else you need around this issue? Or can we close?

[pedastrian57]

@abellina Yes, I am experimenting with different tuning parameters currently, I will create another issue if I get any other doubts about that. We can close this issue. Thank you.