Module Area using 240 MiB on startup - Nightly Pi Build

Hi guys, I have just upgraded from the 23rd July 2016 nightly build to 21 September 2016 nightly build, and noticed the Module Area reading 241300K, sliding the bar down fails so…

How do I list how much each module is using?

I can’t see any out there numbers in the workspace for each module under *modules.
.
I am using a Pi B Mk 2 (not a Pi2) so has 512 MiB RAM

I am still using the !Boot from the old ROM…if that may be causing anything?

Returning to the old ROM shows Module Area usage at 4 MiB (as expected)

Any ideas?

I have just confirmed that it is in the !Boot structure by renaming it to !BootO and installing the !Boot from the latest nightly beta.

Module Area is back down to 2048K

So there is something from the old structure which is whacking out 140 MiB.

If anyone has any idea how to work out (without spending ages removing things and re-adding them) what is causing this, I can investigate in-case others run into this.

For now I will use the new structure (obviously)

Module Area reading 241300K

Less than 4000K here using the same ROM but !Boot 1.25 (16-Sep-16).

Verma may help.

I’d add the things I actually want to use, as and when I want to use them. If you want to investigate for the fun of investigating (or for the benefit of others) then add about 50% of everything initially, then if all’s well add another 25% or if not take 25% away, etc. – that is, do a binary chop to find the culprit. Unless of course someone else can think of a subtler approach!

Found it.

Turns out it is Fat32FS reading the 2 TiB USB Drive I had connected to the hub (forgot to unplug it!)

Oops!

i was going to remove the above, but might as well leave it here in case anyone else falls into this situation.

I was sure it used a dynamic area for it, if it had, I could have easily seen it. Still good to know it can access the 2 TiB Fat32 partition!

so (2TB/32768)*4=256MB.

Damn!

I didn’t realise that FAT32 volumes could be that large, so I did some reading. It seems to be one of those oddities where it’s theoretically capable but not “supported” – Windows (which is of course the native implementation) will only format up to 128 GB, and in the NT family it’s arbitrarily limited to 32 GB to try to push people over to NTFS.

I used Fat32Format under RISC OS to get it to that size…or was it linux? Windows refused to format it unless it was NTFS

might need to move onto my Pi 2 or 3 then!

In it’s development state, my app is using a few maps of the data read frmo said disc, and requires 120 odd Meg (only temporary)

Well the newer DOS implementations have no trouble formatting up to a 2TB volume in FAT32 (the DOS implementations released since 2002 that support FAT32 (including FreeDOS).

OK, but by “support” I mean that Microsoft – the company behind the format – provides tools with a 128 GB limit. Bringing third-party software into the mix takes us back to square one.

But in any case, as far as I know there isn’t a “better” format with cross-OS interoperability. Without RISC OS in the mix I’d use exFAT, but as far as I can tell nobody’s written an “exFATFS” for RISC OS.

We did have ext2 support in RISC OS via an image filing system called IscaFS. I think the source is available if you search. I’m not sure whether it was 32-bitted, but I used to use it regularly to read the ARMLinux partition on our Risc PC.

I’m not sure whether it was 32-bitted

It would be great.

Anyway, FAT32FS is a good way to use big discs. There is probably room for new things, as smaller use of RAM, RISC OS filetype support (,fff), front-end to open several partitions per disc, etc.

Support for big discs is really needed now and Fat32FS could be one of the solutions.

We did have ext2 support in RISC OS via an image filing system called IscaFS. I think the source is available if you search. I’m not sure whether it was 32-bitted

32 bitted by Marco Faye up to version 0.17 with some google

… so larger physical drives are not a problem.

No. Riscos only has a 29 bit address for the sector number on the drive for the DiscOp command.
The upper three bits are the drive number.
The DiscOp64 sys call is actually routed through to the 29 bit command making it totally useless.
Fortunately SCSIFS has a partition command which FAT32FS uses to dynamically select a 256MiB “window” onto the drive. This gives an effective 32bit sector address. So the max size of USB drive is 2T.

Well I think the Fat32fs needs updated to not cach all of the FAT. It is a nice speed up, though not realy of big use. It would better to cache the cluster chains of open files, or maybe of all files in current open dirrectories (more like what newer implementations of DOS do).

It already does exactly that.

Is it possible for the DiscOp’s to be updated for larger Discs?

Can Fat32fs be used on a Iyonix to read a FAT format Drive?

On Adrian’s web site – he talks about his C Stubs – that allows a module program to be included in the RO Rom.

IscaFS 0.17 leads to a dead link :-(

IscaFS 0.17 leads to a dead link :-(

See: https://web.archive.org/web/20110719101733/http://www.esw-heim.tu-clausthal.de/~marco/smorbrod/IscaFS/

Jeff: Can I just say a BIG thanks for FAT32FS!

Is there a reason why you cannot use Dymanic Areas to store the data?

Only reason I would ask this is that you could see it listed in that section, and idiots like me wouldn’t make that mistake again :D

Also do you know why the area did not shrink back when I rmkilled fat32fs?

Turns out I had disabled DosFS? in the older ROM to stop an issue it had with my big disc, and Fat32FS wasn’t auto-loading the drive, but as soon as the new ROM was placed in…bang it pre-initialised… just wish I could change the name of the thread as it is now misleading :)

Just be aware that with FAT32FS, there is some sort of issue where trying to load a potentially damaged file (damaged FAT? dir?) will completely stiff the machine. Not done any poking around to figure out where/why, it’s a little hard given what happens. :-(

Is there a reason why you cannot use Dynamic Areas to store the data?

I just used ‘malloc()’ which is far easier. That way the Clib sorts it all out.

Also do you know why the area did not shrink back when I rmkilled fat32fs?

No I don’t. The module should’ve tidied up when it got the finalise message.

… will completely stiff the machine …

They’re always buggers to find.

Also do you know why the area did not shrink back when I rmkilled fat32fs?

No. The module should’ve tidied up when it got the finalise message.

It’s probably fragmentation. The RMA can only shrink if there’s empty space at the end of the heap; it can’t do anything about empty space in the middle.

Also do you know why the area did not shrink back when I rmkilled fat32fs?
The module should’ve tidied up when it got the finalise message.

It’s probably fragmentation. The RMA can only shrink if there’s empty space at the end of the heap; it can’t do anything about empty space in the middle.

Ah! I had loaded more modules (Reporter most likely)

Edit – how do you nest bq. like that?

Not just loading modules, but any RMA claim that isn’t released.

Your hair would raise if you knew what was going on in the RMA: https://www.heyrick.co.uk/blog/index.php?diary=20160801

32 bitted by Marco Faye up to version 0.17 with some google

Cool, thank you! Just need a tool to format the disc from RISC OS :)

RISC OS filetype support (,fff)

Should be automatic for all non ADFS/SCSIFS/ETCFS filesystems.

Edit – how do you nest bq. like that?

You cheat and use HTML <blockquote> tags :-)

That’s also how to get blank lines in a blockquote, I found.

32 bitted by Marco Faye

Minor detail, but the name on the web site is Marco Baye.
Typo perhaps? (I think he may know how to spell his name :) so he won’t get it wrong)