Regarding GPU flushes

[mysoreanoop]

Describe the bug

1. What is the reason F state doesn't have any possible transitions for other events while in F state -- I'm assuming it would be possible to get a requests from CPU caches at least, right?
2. TCC is not sending flush response (to TCP) when block-in-question is at V/W state. It does, however, send the response when the line is at I state. Is there a reason for this?

Tagging @v-ramadas based on commit 085789d

Affects version
Latest stable and develop

Additional information
There are no failing test cases so far, I'm just unable to logically reason about this. It is, however, important for my work.
@mattsinc Please let me know if these kinds of issues need to be taken elsewhere, I'd be happy to oblige :)

[v-ramadas]

Hi @mysoreanoop,

1. Since most data cache flushes happen right before a checkpoint is taken, I assumed that the a flush is the last event before end of simulation. I haven't seen any other case in the applications I've run so far (Pannonia, heterosync, and square). Let me know if this reasoning doesn't hold true in all cases, and I will add a stall for other events (When line is in F state)
2. If a line is in V,W state in the TCC and it receives a flush request, it sends the cache line data to the directory before invalidating itself:

   gem5/src/mem/ruby/protocol/GPU_VIPER-TCC.sm

   Line 1130 in 0442c9a

   f_flush;

   
   The directory flushes this data down to memory and returns a WBAck response: https://github.com/gem5/gem5/blob/0442c9a88c5e0031d766470c76d1cd8138b313cb/src/mem/ruby/protocol/MOESI_AMD_Base-dir.sm#L1601C7-L1601C7
   The TCC sends an ack to the TCP after receiving the WBAck from the directory:

   gem5/src/mem/ruby/protocol/GPU_VIPER-TCC.sm

   Line 1104 in 0442c9a

   w_sendResponseWBAck;

   
   The TCP then completes the flush by notifying the GPUCoalescer:

   gem5/src/mem/ruby/protocol/GPU_VIPER-TCP.sm

   Line 759 in 0442c9a

   f_flushDone;

   
   The flush request is marked as completed only after the GPUCoalescer receives this acknowledgment. I haven't run into any failing test cases so far but would be interested to see if there is something that breaks this.

Please let me know if this doesn't make sense or if you run into any failures or deadlocks - I'd be happy to take a look.

[BobbyRBruce]

I moved this to discussion Q&A as I don't think this is actually reporting a bug, simply asking about gem5 behavior.

[mattsinc]

+1, thanks @BobbyRBruce

[mysoreanoop]

@v-ramadas
Thank you very much on the detailed explanation on point 2. Completely makes sense!
A small followup:
https://github.com/gem5/gem5/blob/0442c9a88c5e0031d766470c76d1cd8138b313cb/src/mem/ruby/protocol/MOESI_AMD_Base-dir.sm#L1600C1-L1600C1 -- here, we write back the flush data to memory without probe-invalidating any sharers (at least on the CPU side?) -- is this safe?

[v-ramadas]

Sorry, I missed this follow-up entirely. I've been using a KVM-CPU in my tests in FS mode and this doesn't use any of the ruby protocols. I believe what this means is that flushing (and checkpointing by extension) is known to work in a KVM-CPU + GPU system in FS mode. I guess this is part of a move towards making the FS mode a viable alternative. That being said, I would be happy to take a look at the CPU side for other CPU models once I have a system up and running.

[mysoreanoop]

I see. I will try to understand that and get back to you, I think I've been thinking about flushes wrongly.

[mysoreanoop]

I'm not sure I understand (1) -- I'll think about it and get back to you in a few days, sorry.