posts - 20 , comments - 57 , trackbacks - 0

PCI Latency Timer

In my previous blog I mentioned I was involved in searching for Windows socket data that got corrupted upon reception in a Windows CE 6 executable.

The short explanation is that the Realtek 100MBit (RTL8139, RTL8101) network interface card simply couldn’t swallow high load bursts of TCP packets (The long explanation took weeks...). The Realtek card couldn’t transfer its data fast enough to the cpu’s main memory for further handling by NDIS and the Windows CE 6 TCP/IP stack. The loss of data showed itself on the wire (WireShark) as TCP RETRANSMISSIONS and when applicable as TCP FAST RETRANSMISSIONS.

What was happening? Well the NIC acts as a PCI Bus Master and therefore the PCI Latency Timer in the PCI configuration register comes into place. The PCI Latency Timer determines how long a PCI Bus Master can stay on the PCI bus before relinquishing access to the bus (in favor of other bus masters requesting access). This value is by convention 32 [0 .. 255] bus clock cycles and set by the BIOS for each PCI bus master.

The BIOS of the SBC that we were using however set the value to 0 (zero!). 0 means relinquishing access immediately when another bus master requests access. With the result that sometimes on high loads TCP packets got lost. By setting the value to 32 the problem disappeared.

The BIOS value of 32 is a good default value and given to each PCI bus master present on the bus. On rare very high load conditions however, still packets got lost. Therefore I wrote a small piece of code that was executed by the Windows CE image at bootup (in oeminit.c) that sets the value to 32 for most PCI bus masters, but to 64 for Ethernet network cards. This fine-tuning completely solved the problem.

void FixupPciLatencyTimer(void) { USHORT deviceId; USHORT command; USHORT devId; BYTE latencyTimer; BYTE classCode; BYTE subClassCode; BYTE b, d, f; BOOL isBusMaster; PCIReadBusData(0x0, 0x1F, 0x0, &deviceId, 0x02, 2); RETAILMSG(1, (TEXT("fixupPciLatencyTimer LPC deviceId = 0x%x\r\n"), deviceId)); for (b = 0; b < 255; ++b) // bus { for (d = 0; d < 32; ++d) // device { for (f = 0; f < 8; ++f) // function { PCIReadBusData(b, d, f, &devId, 0x02, 2); if (devId != 0xFFFF) { PCIReadBusData(b, d, f, &classCode, 0x0B, 1); PCIReadBusData(b, d, f, &subClassCode, 0x0A, 1); PCIReadBusData(b, d, f, &command, 0x04, 2); isBusMaster = (command & 0x0004) == 0x0004; RETAILMSG(1, (TEXT("%02x %02x %01x %02x %02x : cmd = %04x %s\r\n"), b, d, f, classCode, subClassCode, command, isBusMaster ? TEXT("BusMaster") : TEXT("-"))); PCIReadBusData(b, d, f, &latencyTimer, 0x0D, 1); RETAILMSG(1, (TEXT("%02x %02x %01x %02x %02x : latTim = %02x\r\n"), b, d, f, classCode, subClassCode, latencyTimer)); if (isBusMaster /*&& latencyTimer == 0x00*/) // It is possible that Latency Time is ReadOnly { if ( classCode == 0x02 && // class = 0x02 = network controller subClassCode == 0x00) // subclass = 0x00 = ethernet controller { latencyTimer = 64; // good default value. Required for Realtek 100MBit nic cards. } else { latencyTimer = 32; // good default value. } PCIWriteBusData(b, d, f, &latencyTimer, 0x0D, 1); // write PCIReadBusData (b, d, f, &latencyTimer, 0x0D, 1); // read back RETAILMSG(1, (TEXT("%02x %02x %01x %02x %02x : set latTim = %02x\r\n"), b, d, f, classCode, subClassCode, latencyTimer)); } } } } } }

The attentive reader however might argue (correctly) that TCP packets should never get lost. TCP checksum and retransmission mechanisms should guarantee errorless transmission of data.

Well, this was not the case. Therefore I believe there is still a bug in the TCP FAST RETRANSMISSION mechanism in the Windows CE 6 TCP/IP stack. When TCP FAST RETRANSMISSIONS were logged on the WireShark traces, we sometimes saw in our application the wrong data been delivered over the Windows socket.

As I could circumvent the problem by setting the PCI Latency Timer to a reasonable value (64), I stopped debugging it. However this code will be converted in the future to Windows CE 8 and I will certainly pick it up again then. The Windows CE 8 TCP/IP stack is based on the desktop Windows 7 TCP/IP stack while the Windows CE 6 TCP/IP stack is based on desktop Windows XP TCP/IP stack. Different enough to see if there is also a difference in behavior when the PCI Latency Timer is set back to 0.

We also tested the same PCI Latency Timer = 0 on an Intel PRO 100 network card, but that card always behaved correctly. Even with a value of 0, no data packets were lost, nor were TCP (FAST) RETRANSMISSIONS visible on the wire.

Print | posted on Tuesday, July 1, 2014 9:29 PM | Filed Under [ Windows CE Windows Embedded Compact Microsoft NDIS TCP/IP ]


No comments posted yet.
Post A Comment

Powered by: