PCI heap conundrum

On startup the PCI module looks for a physically contiguous block of memory around 32MB in size for it to use as the PCI heap. Once it’s found the region, it remembers the base page number, and then requests appropriate page numbers from within that region on the DA PreGrow calls.

This works fine, so long as there are no other components in the system which are interested in specific physical pages. E.g. if you had a second module which attempted to implement a physically contiguous heap in the same way as the PCI module, there’s a good chance they’ll both end up fighting over the same pages, potentially blocking one of the heaps from growing. This isn’t a problem at the moment (the only other physically-contiguous component in most systems is the kernel-managed video memory, which the PCI module knows to avoid), but could become a problem as soon as video drivers are given the ability to allocate and manage their own memory.

Potential solutions I can think of:

• Add some ability for programs to ‘earmark’ pages for future use. I.e. the PCI module will indicate that it wants to use those pages, but that it doesn’t need them right now. This would allow regular DAs to make use of the pages (since they can cope with the kernel replacing the pages if the PCI module later decides that it does want to use them), but would prevent the pages from being suggested for physically-contiguous use by OS_Memory 12 or similar. There are a few ways this could be handled:
  • A couple of new calls to earmark/un-earmark memory (programs specify the pages themselves)
  • A service call (programs can override the OS_Memory 12 results – may be tricky to coordinate if the OS_Memory 12 region crosses over several regions which have been earmarked by multiple programs)
  • A more hands-on approach to physical memory allocation by the kernel, e.g. using the buddy block system to carve physical memory into chunks and hand it out to programs/DAs. This could also be used as a way of avoiding some of the pitfalls with OS_Memory 12 itself (since the OS_Memory 12 result is only valid if the next page grow operation is for your DA, it doesn’t lend itself very well to multi-threaded scenarios, or if we were to start allowing DAs to grow from within IRQ handlers)
• Make the PCI module detect when one of the pages has been claimed by someone else, and search for a new physically contiguous block. This would complicate the module a bit since it would now have to keep track of several physical areas, and depending on how many other modules are running their own physical heaps it could end up in a fair amount of both logical and physical memory fragmentation.
• Change the PCI heap to be a shrinkable DA, and have it grow to max size on startup. This will prevent OS_Memory 12 from recommending the pages to other DAs, at least until the free pool runs out of pages, at which point the kernel will make a shrinkable shrink request and the PCI module will return at least some of the pages back to the free pool (Or the user could drag the PCI heap size down via the task manager. Hmm. Maybe we should render shrinkable areas differently, so it’s clear that the DA isn’t hogging memory?)
• When video drivers gain control over their memory management, make the kernel be in charge of which pages the drivers use. This will reduce the amount of fighting between drivers and the PCI module, and could allow the kernel to de-frag physical memory if necessary. But it would perpetuate the special VRAM memory block that exists in the kernel (and is only 32MB in size!), and won’t do much to help anything else that might want to manage its own physical memory. (Maybe the de-fragging doesn’t need to be specific to video memory? A SWI which attempts to de-frag all physical memory could send round a service call to collect a list of de-fraggable blocks, and then attempt to coalesce them as much as possible, coordinating the operation with each block owner. Although with modern systems sporting >2GB RAM it’s unlikely we’ll have to worry about physical memory fragmentation, as long as systems aren’t constantly fighting over the same pages)

Any other options people can think of?

I might be stupid here but…

• Why does the PCI module need a specific contiguous block for its heap ? For direct hardware access to chunks of memory, I could understand, but for its heap?

And…

• Surely if something else (video driver or whatever) also wanted specific physical pages, wouldn’t it be requesting different addresses (thus being allocated different pages)?

I feel like I’m missing something…?

Why does the PCI module need a specific contiguous block for its heap ? For direct hardware access to chunks of memory, I could understand, but for its heap?

The heap is used to implement PCI_RAMAlloc, which is used by assorted drivers which want easy access to physically contiguous memory (and don’t have any constraints like e.g. needing to remap the memory in logical space, or needing to deal with giant video framebuffers which might overflow or seriously fragment the PCI heap)

Surely if something else (video driver or whatever) also wanted specific physical pages, wouldn’t it be requesting different addresses (thus being allocated different pages)?

How can it, if it doesn’t know which pages have been “claimed” by the PCI module?

E.g.:

1. PCI module starts up, and calls OS_Memory 12 to locate 1000 pages of physically contiguous memory
   1. OS_Memory 12 returns, saying that pages 1000-2000 are free
   2. PCI goes “great, I’ll remember that!” and makes a note of that region in its workspace
2. USBDriver asks for some memory via PCI_RAMAlloc
   1. PCI grows its heap a bit, so that it now contains pages 1000-1003. Pages 1004-2000 are still free for anyone to use
3. BobDriver starts up, and asks OS_Memory 12 to locate 100 physically contiguous pages
   1. OS_Memory 12 says pages 1004-1103 are free, since they’re the lowest-numbered pages which haven’t been claimed yet
   2. BobDriver immediate creates a DA and claims all those pages for itself
4. USBDriver comes along again, and asks for some more memory from PCI_RAMAlloc
   1. PCI tries to claim pages 1004-1007, but it can’t

Aha. I see. So PCI doesn’t want to claim pages it doesn’t need, but wants to lay claim to pages it hasn’t allocated for itself.
Sounds like you need a PreAlloc flag, to inform whatever that while the page isn’t actually allocated, something has laid claim to it.
Or, in your list, the earmarked option. ;-)

Thinking about this earlier today, using a service call to influence the OS_Memory 12 results isn’t going to be a very good solution for threaded scenarios, since the act of reserving memory won’t be atomic (two concurrent calls to OS_Memory 12 could return the same block of memory to the two different clients, because those clients are only able to update their service call handlers to report the memory as being reserved once the corresponding OS_Memory 12 call has completed)

So it would probably make sense to go with an explicit reserve/unreserve SWI pair, so that looking for a block of memory and marking it as reserved can be fully atomic. The reserve SWI could be similar to OS_Memory 12 (in that the kernel is able to do most of the work to determine the address range to use), but extending it to support a function callback would also be nice (so programs can fully implement their own logic for determining which range of pages to reserve).

Since we want the reserved (but not-yet-used) memory to be made available for use by non-fussy DAs/PMPs, the reserved memory probably won’t have any DA or PMP associated with it, in the traditional sense. But maybe it would make sense to have some kind of association just so that the kernel can keep track of the memory more easily. Or maybe if we want to stop pages being poached by programs that didn’t reserve them via the official API (e.g. you can only lock that page in your DA/PMP if you can prove that you’re the one who reserved it).

So it would probably make sense to go with an explicit reserve/unreserve SWI pair

This is the approach that was taken, back in 2019.

https://gitlab.riscosopen.org/RiscOS/Sources/Kernel/-/commit/1f84ad9f0a7089821d039601527141e98598a47c
https://gitlab.riscosopen.org/RiscOS/Sources/HWSupport/PCI/-/commit/360c9fc5baad07b474289380b328b1ac7c7bd3ce