Scripts to use all the Pi SD Card

I know partition support is “on the way” but I have knocked together some scripts to use all the SD card on a RPi.

Not for the feint hearted…

It’s a somewhat manual process of

Copy the img file to an SD card
Use dd to extract the DOS bit to a file
Boot into RISC OS, reformat the SD card
Put it back in the linux machine

The scripts are the used to:

find where to put the DOS bit, and put it there (it depends on the LFAU, as the first LFAU is used for the the Disc Record)
re-write the MBR to get the parition table back
Reserve the area used by the DOS area in the FS Map
Create the ‘Loader’ file (in $)

Then, copy the self-extracting hard disc image into the DOS area

Put the card back in the Pi, and boot it.

use *Desktop to enter the desktop, open Loader, copy the self-extractor to $ and run it.
Delete the copy of the sfx, and move Loader into !Boot

Reboot

You’ll now have the basic hard disc image, but not the Pi extras. They’re too bit to fit in Loader, so I’d suggest copying them onto a RISC OS format USB stick (or zipping them onto a DOS format one) from the original Pi system before you format the card.

If it’s of any use to anyone I can try to tidy them up a bit, but I can make no guarantees and it’s really an ‘expert users only’ thing. DiscKnight says the resulting disc is ‘good’, but again … use at your own risk.

Chris,

These do sound interesting – any chance of a look?

I have put the scripts up on github. They are in python but don’t need anything other than the very standard stuff.

There are some notes on the Pi resizing.

The project is (badly) named as adfs-tools – https://github.com/c-jo/adfs-tools

Let me know how you get on, or if I’ve missed something.

Good luck!

Thanks Chris – I’ll try them out and let you know how I get on.

Cheers. Just remember to backup, and then backup the backup of anything on the card if you’re not making one from the downloaded image.

Also it assumed the SD card will be /dev/mmcblk0 in Linux, so check that it is :)

Does the utility mark the filecore partition as such? (There is a partition type code registered for filecore.)

It creates two partitions – the first the DOS one and the rest of the disc is marked as filecore (ad). I guess technically since filecore doesn’t yet understand partitions so uses all the disc, so that’s not actually correct, but it at least stops others systems splatting over the data.

With two overlapping partitions, marking the extent of each is always going to be a compromise…

Chris J,

Just a quick update to let you know that I’ve finally gotten around to trying this out and it all seems to work as described.

In the end, I used an Ubuntu VM running in Virtual Box on my Windows machine with a USB card reader directly attached to the VM.

As you mention, it was necessary for me to change the path to the device file representing the attached card reader – in my case, this was ‘/dev/sdb’ – you can see the changes I made in my fork on GitHub.

I started off by creating 2 SD cards from the 5.24 Pi download image – one as the target card, the other for the source of the original files to copy across.

Now to get on with filling up the 32gb SD card…

Thanks again for putting these scripts together.

Glad they worked for you. I should probably make the device a parameter to the script rather than hard coded, I’ll try to get that when I get chance.

I’ve had a few cracks at this and keep ending up with an unbootable SD Card. I had to add an “Import sys” line at the start of each script to get them to run using Python2 on my Ubuntu laptop. The scripts look like they’re working as intended when run at the command line and extract a Loader file from the disc image. I think I managed to get HForm to format the SD card -it shows up as 30GB. However, if you run any kind of soak test then Hform hangs at the verify stage. Only running without verification lets the formatting complete.

After running all the scripts the resulting SD card doesn’t boot at all. The Pi just sits with the red power light on and no other lights. It boots up fine using the standard Risc OS image.

Anyone got any ideas about where the issue lies?

It creates two partitions – the first the DOS one and the rest of the disc is marked as filecore (ad). I guess technically since filecore doesn’t yet understand partitions so uses all the disc, so that’s not actually correct, but it at least stops others systems splatting over the data.

Is comparable to something I discribe (and Chris Gransden on a different way) any years ago (2012) in this forum.
But SystemDisc is in my eyes an easier way and not really expensive.

But SystemDisc is in my eyes an easier way and not really expensive.

I agree.

I might well end up investing in a copy of SystemDisc, but would still like to get to the bottom of why this isn’t working. Could it be that the SD card isn’t actually being formatted? I booted up the Raspberry Pi from the 32GB card with the standard 2GB RISC OS image on it. Then I ran HForm from that card and used it to format the whole card with FileCore. Should that work?

Should that work?

Depends. If you accepted the current format values, then No.

I answered no to the question about retaining the current shape and then accepted HForm’s suggested values for cylinders, tracks etc. I was just curious about whether there would be an issue with using HForm to format the drive that it and the operating system is stored on.

I can assure you that HForm will happily format your main drive – it wiped my 128GB SSD when I asked it to format drive 0 having unfortunately answered the default ‘S’ for SDFS (which should, of course, be ‘M’) when I was dropped into HForm by SystemDisc, assuming it would specify the right default parameters for me. Unfortunately SystemDisc is not that well written. I also knew my SSD was drive 4 not 0 but it was the first SCSI device enumerated and so HForm referred to SCSI::4 as drive 0. HForm has been improved a bit since then…

Then I ran HForm from that card and used it to format the whole card with FileCore. Should that work?

Hum… no. You’ll erase the FAT partition reserved for the Pi boot files (and your RISC OS partition too).

The good way to do this could be to format only a part of the disk with HForm. Then to switch to Linux and add a small FAT16 partition to the end of the SD Card (size = size of the SD card – size of the SDFS partition – a few MB for security). And then to copy the boot files in the FAT16 partition and the RISC OS files in the SDFS partition.

After you have formatted your SD card with Hform, you will need to run SystemDisc to create the overlapping FAT partition and then copy all the RISC OS files (except Loader) back and then copy back the FAT partition firmware.

having unfortunately answered the default ‘S’ for SDFS (which should, of course, be ‘M’)

You can reduce the likelihood of that problem with an edit to !HForm.Messages. Approximately 4 lines from the end of the file is the line
WhichFSSM:Is your drive connected to SCSIFS or SDFS (S/M) ?

Change that to read:
WhichFSSM:Is your drive connected to SCSIFS or MicroSD (S/M) ?

I went the extra mile and changed the question to:
WhichFSSM:Is your drive connected to USB or MicroSD (U/M) ?
but (a) that’s only suitable for the Raspberry Pi, and (b) you have to change another line (the one that reads SM:SM – in HForm 2.75 it’s the last line of the file) to accept U and M as responses, i.e. the line becomes: SM:UM

You can reduce the likelihood of that problem with an edit to !HForm.Messages.

This simple change to the Messages file would be the work of a moment in the source. Could someone (Sprow?) not just “do it”! “Micro SD” is so obviously what was in mind when the programmer chose “M”!

I had been wondering why M stood for SDFS. It’s in part why I managed to format the SSD in my ArmX6. Not the best thing I’ve ever done.

I’ve been playing around with the scripts a bit more. It seems that they succeed to some degree in creating partitions on my SD card. Ubuntu’s Disks app shows a 134MB FAT32 partition followed by a 31GB other partition. However, it isn’t possible to mount the FAT partition. Could this be evidence of where the problem lies? Possibly some incompatibility between the approach taken to format the card and my Ubuntu set up?

There’s a script called “explore” that shows the various SDFS folders that I copied back onto the SD card after I formatted it with HForm. It also shows the Loader file in the root directory where it should be. I think the RISC OS side of things seems to have worked pretty well.

I’ve had another look at this. The script that does the formatting of the DOS partition is called put_loader.py. I’ve copied the contents below in the hope that someone who understands Python can decipher what’s going wrong.

When I run it, it reports a LFAU of 42K. This compares to the 2K LFAU suggested by HForm for the SDFS partition. The size of the DOS partition that put_loader creates is 134MB. In the standard 2GB SD card image the DOS partition is only 50MB, 40MB of which is empty.

Can anyone see what the issue is with the script below? I added the “import sys” at line 4. The script wouldn’t run without it.

#! /usr/bin/python

import struct
import sys
from utils import get_map

DOS_MAX = 128*1024*1024

if len(sys.argv) != 2:
print(“Usage: put_loader ”)
exit(1)

fd = open(sys.argv¹, “rb”)
dos_data = fd.read(DOS_MAX)
fd.close()

print “DOS area is {0} bytes.”.format(len(dos_data))

fd = open(“/dev/mmcblk0”, “r+b”)

fs_map = get_map(fd)
lfau = (fs_map.disc_record.idlen+1)*fs_map.disc_record.bpmb

dos_start = lfau / fs_map.disc_record.secsize
dos_secs = len(dos_data) / fs_map.disc_record.secsize

print “Disc has LFAU of {0} ({1}K), so loader area starts at sector {2}”.format(lfau,lfau/1024,dos_start)

fd.seek(0, 2)
disc_size = fd.tell()

adfs_start = dos_start+dos_secs+1
adfs_secs = disc_size / fs_map.disc_record.secsize – dos_secs

fd.seek(0, 0)

chs_dummy = ( 254, 255, 255 )
part_table = fd.read(512)

p1 = part_table[0×1be:0×1be+16]
p2 = part_table[0×1ce:0×1ce+16]
p3 = part_table[0×1de:0×1de+16]
p4 = part_table[0×1ee:0×1ee+16]

p1_data = struct.unpack(“BBBBBBBBII”,p1)
p4_data = struct.unpack(“BBBBBBBBII”,p4)

p1_new = struct.pack(“BBBBBBBBII”, 0×80,\
chs_dummy⁰,chs_dummy¹,chs_dummy², 11,
chs_dummy⁰,chs_dummy¹,chs_dummy²,
dos_start, dos_secs)

p2_new = struct.pack(“BBBBBBBBII”, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
p3_new = struct.pack(“BBBBBBBBII”, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )

p4_new = struct.pack(“BBBBBBBBII”, 0×00,\
chs_dummy⁰,chs_dummy¹,chs_dummy², 0xad,
chs_dummy⁰,chs_dummy¹,chs_dummy²,
adfs_start, adfs_secs)

new_part = part_table[0:0×1be] + p1_new + p2_new + p3_new + p4_new + “\x55\xaa”

fd.seek(0, 0)
fd.write(new_part)

fd.seek(dos_start * fs_map.disc_record.secsize, 0)
fd.write(dos_data)

fd.close()

The variable lfau in that script is what I would call “minimum fragment size” (ignoring rounding – see PRM 2-206).

What is usually called lfau is fs_map.disc_record.bpmb.