PCI probing takes 4 time units in qemu

[shenki]

Running @legoater 's qemu tree (3.1 based) with the powernv model:

[    0.270844220,5] PCI: Resetting PHBs and training links...
[    0.272165254,7] PHB#0000: FRESET: Assert skipped
[    0.272202593,7] PHB#0000: FRESET: Deassert
[    1.272533870,7] PHB#0000: LINK: Start polling
[    1.324060833,7] PHB#0000: LINK: Electrical link detected
[    1.376168984,7] PHB#0000: LINK: Link is up
[    1.377293537,7] PHB#0001: FRESET: Assert skipped
[    1.377331212,7] PHB#0001: FRESET: Deassert
[    2.377747704,7] PHB#0001: LINK: Start polling
[    2.429170789,7] PHB#0001: LINK: Electrical link detected
[    2.480562886,7] PHB#0001: LINK: Link is up
[    2.480931339,7] PHB#0002: FRESET: Assert skipped
[    2.480958483,7] PHB#0002: FRESET: Deassert
[    3.481109850,7] PHB#0002: LINK: Start polling
[    4.020507587,7] PHB#0002: LINK: Electrical link detected
[    4.071916279,7] PHB#0002: LINK: Link is up
[    4.072493625,5] PCI: Probing slots...
[    4.073907375,7] PHB#0000:00:00.0 Link up at x1 width
[    4.073995548,7] PHB#0000:00:00.0 Scanning (upstream+downsteam)...
[    4.076130479,7] PHB#0000:00:00.0 Found VID:1014 DEV:03dc TYP:4 MF- BR+ EX+
[    4.083441488,7] PHB#0000:00:00.0 Bus 01..ff  scanning...
[    4.101693656,7] PHB#0001:00:00.0 Link up at x1 width
[    4.101785886,7] PHB#0001:00:00.0 Scanning (upstream+downsteam)...
[    4.102132269,7] PHB#0001:00:00.0 Found VID:1014 DEV:03dc TYP:4 MF- BR+ EX+
[    4.105512107,7] PHB#0001:00:00.0 Bus 01..ff  scanning...
[    4.112340762,7] PHB#0002:00:00.0 Link up at x1 width
[    4.112368924,7] PHB#0002:00:00.0 Scanning (upstream+downsteam)...
[    4.112476835,7] PHB#0002:00:00.0 Found VID:1014 DEV:03dc TYP:4 MF- BR+ EX+
[    4.113921390,7] PHB#0002:00:00.0 Bus 01..ff  scanning...
[    4.118325244,5] PCI Summary:
[    4.118992280,5] PHB#0000:00:00.0 [ROOT] 1014 03dc R:00 C:060400 B:01..01 
[    4.119569928,5] PHB#0001:00:00.0 [ROOT] 1014 03dc R:00 C:060400 B:01..01 
[    4.120173134,5] PHB#0002:00:00.0 [ROOT] 1014 03dc R:00 C:060400 B:01..01 

We should see if the model, or skiboot, can be modified to reduce this time.

[oohal]

That log says that PCI probing is pretty much instant:

[    4.072493625,5] PCI: Probing slots...
...
[    4.118325244,5] PCI Summary:

The time consuming part is resetting each PHB and training the links. IIRC there's a bunch of mandatory waits in there so if you're booting with a single CPU it'll be slow.

[ghost]

Yeah, it'll be slow because of those compulsory waits.
Arguably we should do them all at once though and take one unit of waits rather than several.

[oohal]

So I was looking at this today and we might be able to make it a bit faster. Currently the PHB3 model always reports that there is in-band presence since the value of the training control register is hard-coded. You can hack that with this:

diff --git a/hw/pci-host/pnv_phb3.c b/hw/pci-host/pnv_phb3.c
index 92828fa2d1b6..79196e28bbf0 100644
--- a/hw/pci-host/pnv_phb3.c
+++ b/hw/pci-host/pnv_phb3.c
@@ -585,6 +585,7 @@ void pnv_phb3_reg_write(void *opaque, hwaddr off, uint64_t val, unsigned size)
 uint64_t pnv_phb3_reg_read(void *opaque, hwaddr off, unsigned size)
 {
     PnvPHB3 *phb = opaque;
+    PCIHostState *pci = PCI_HOST_BRIDGE(phb);
     uint64_t val;
 
     if ((off & 0xfffc) == PHB_CONFIG_DATA) {
@@ -614,6 +615,8 @@ uint64_t pnv_phb3_reg_read(void *opaque, hwaddr off, unsigned size)
 
     /* Link training always appears trained */
     case PHB_PCIE_DLP_TRAIN_CTL:
+        if (!pci_find_device(pci->bus, 1, 0))
+            return 0;
         return PHB_PCIE_DLP_INBAND_PRESENCE | PHB_PCIE_DLP_TC_DL_LINKACT;
 
     /* FFI Lock */

Unfortunately, this doesn't make things any faster right now since there's a 1s timeout when we're waiting for the in-band presence bit to go high. That said, the PCIe spec says that:

a) Devices must enter the link Detect state within 20ms of PERST being lifted, and
b) The transition from Detect to Polling (where in-band presence is 1) should be 24ms.

So we should be able to make it faster. There might be a good reason for the longer timeout though, @mikey or @ozbenh might know why.

[legoater]

Do you want me to merge the patch above in the PHB3 model ?

[oohal]

@legoater Sure, it won't fix the problem here, but it makes the model a little better and shouldn't break anything. The PHB4 model has the same issue too.