Pi 3 shutting down-ish

I’ve upgraded to 5.25 (02-06-18 build) an hour ago and have had over five blank screens since. It’s completely random, one occurred within seconds of reaching the desktop and another occurred whilst I was typing!

It appears that the HEAD branch of BCMVideo used in BCM2835Dev doesn’t have gamma disabled. 5.24 does have it disabled and doesn’t suffer from the problem.

The point I’m probably making is that I can readily reproduce the issue, should we need to do any testing. When it occurs, the GPU appears to lock solid and all audio immediately stops. The keyboard continues to work, as do apps etc.

It’s occurring within minutes of boot, so the OS is completely unusable – but ideal for testing fixes. I’ve also noticed it only occurs when at the desktop, whilst testing full screen games the problem doesn’t occur.

Another thing I’ve noticed, it occurs more often if I’m typing and/or using the mouse!

EDIT: Is this issue being triggered by the spurious palette changes I reported a few years ago? That might explain why I don’t see the issue when testing games, as ADFFS will claim the palette changes, preventing them from reaching the BCM2835 driver.

Is this issue being triggered by the spurious palette changes I reported a few years ago?

No, is the answer however suppressing all palette changes does substantially reduce the number of times the problem occurs. Normally, I see the problem within seconds to minutes of booting. With palette changes suppressed the problem only occurs about once every half hour.

Is the GPU getting overloaded with requests perhaps?

Is the GPU getting overloaded with requests perhaps?

When setting the gamma table we bypass the GPU firmware and update the registers directly (there isn’t a mailbox message for setting the gamma table).

There’s been no progress on the github issue that Sprow opened, so other than completely disabling the code I’m not sure what our options are (last time I tried, I completely failed to reproduce the issue).

There are some known issues with the Pi 3B+ being unstable (apparently caused by some chips deviating from the expected manufacturing tolerances). It’s obviously not the same problem we’re seeing here (since our problem has been seen on non-3B+’s), but it’s something to be wary of when testing.

There’s been no progress on the github issue that Sprow opened

It’s probably worth anyone who sees it and happens to have a github login to chip in with their setup, even if it’s just a “me too”. At the moment it’s only me talking to myself which is likely to mean it’s not getting much attention from the Pi folks. What’s different about Jon’s Pi that triggers it within minutes?

What’s different about Jon’s Pi that triggers it within minutes?

Nothing, that can happen here too, as reported here and here

I rather got the impression that popcornmix did not think much of this implementation :-

Writing to registers that the firmware believes it owns is in the “undefined behaviour” territory – I’d like to know exactly which parts of the HVS are being changed without the firmware’s knowledge.

Moving on :-

(last time I tried, I completely failed to reproduce the issue).

That can happen here but the last time I tried it fell over PDQ. I have to say I do not see the point of continuing to offer a broken Pi OS5.25 development ROM ATM, until a potential fix is available.

Not that I understand any of it but would not a better approach be to ask for a mailbox to do this. After all poking RISC OS directly is asking for trouble, the proper thing is to use system calls.

I have stayed with OS5.24 on my Pi’s but it is simple to build a gamma disabled OS5.25.

Anyway HTH.

I’d like to test if setting the gamma once triggers the issue at some point in the future (ie a GPU issue), or if it’s repeated writing to the registers that’s the catalyst (ie a software or interrupt issue)

When its writing to the GPU are IRQ disabled to prevent clashes with the firmware?

What’s different about Jon’s Pi that triggers it within minutes?

It’s an original Pi3 in a pi-top, not sure on the firmware date, I’ll check when I’m home. It’s completely unusable though, one in four boots results in it going directly to a blank screen as soon as it reaches the desktop.

Going by the random nature, I’d say its a software issue, possibly a clash with the firmware. Its definitely worse when palette changes are occurring and when moving the mouse and using the keyboard. If I leave the machine alone it sits happily at the desktop for ages, within a few minutes of using it however the GPU stiffs.

The fact it doesn’t occur when my GraphicsV driver is active definitely points to something the BCM2835 driver is doing.

EDIT: Firmware is dated 14-12-16 … I should probably update it!

EDIT2: Updating to today’s firmware (03-06-18) has made no difference

EDIT3: Now I’ve had a chance to do more testing with the latest firmware, the problem is worse. Three in four boots to the desktop now result in a stiffed GPU within a few seconds

I have less than 24 hours experience with RISC OS, but does everyone here have a 2.5A minimum power supply as I know at least the 3 B+ will crash if it does not have a properly spec’d power supply.

I use mine on a quick charge 2.0 certified charger. It does provide 2.4A to each port at a much higher wattage than the Pi can handle. I believe the dedicated ones are cheaper but i had this already lying around. If you are buying one just for the Pi and nothing else you can save half the money and get a 2.4a Pi supply. https://www.amazon.com/gp/product/B01BBZJ31Y/ref=oh_aui_search_detailpage?ie=UTF8&psc=1

This particular issue isn’t power supply-related; the system is perfectly stable with gamma disabled.

It’s probably worth anyone who sees it and happens to have a github login to chip in with their setup, even if it’s just a “me too”.

I’ve finally done this, to an extent (I haven’t put any specifics around my setup because I don’t really understand the problem well enough!). Nobody else has said anything though :(

Nobody else has said anything though :(

I have now.

I had a brief look at the Gamma blanking issue today and would say it’s possible its hardware related. It’s either a “feature” of the VC4 or more likely, there’s not enough bandwidth for the HVS to keep the output buffer full, so it shuts down.

There’s a couple of ways to check if bandwidth is an issue, such as lowering both the input and output resolutions or increasing the CPU/GPU/RAM speed – I’m not sure which is relevant to the HVS.

Documentation on the HVS would be helpful at this point.

Documentation on the HVS would be helpful at this point.

I did find this which may or may not help.

It does introduce dispmanx_offline=1 which is new to me.

My RPi3B+ is back on the case with the latest firmware and OS5.25 with that line added. I don’t immediately see any difference in the Pi’s performance and the screen refresh rate is still 60Hz.

(Or is this just another way of disabling gamma!)

You’ll only see it drop below your output refresh rate if the display can’t be composited within the one VSync, if bandwidth is the issue we’re seeing then it will be vary rarely, given the amount of time it’s taking some folk to get the screen to blank.

As I can get it to blank within minutes, I’ll see if offline compositing makes any difference.

dispmanx_offline=1 makes no difference, I’m still getting a blank screen within seconds of reaching the desktop.

Dropping the desktop resolution to 800×600×16M does however appear to improve the blanking, so it certainly hints at being bandwidth related. Going to 1024×768×16M it blanks within seconds.

If its of relevance, the resolution I have configured in CONFIG/TXT is:

hdmi_cvt=1360 768 50 3 0 0 0
hdmi_group=2
hdmi_mode=87

Which is a custom resolution for the pi-top screen @ 50Hz

If you want to test Gamma on RISCOS 5.24, I knocked up the code below to enable it:

IO_BASE=&3F000000
IO_BASE_SIZE=&1000000
SCALER=&400000
SCALER_DISPCTRL0=&40
SCALER_DISCBKGND0=&44
SCALER_DISPCTRL1=&50
SCALER_DISCBKGND1=&54
SCALER_DISPCTRL2=&60
SCALER_DISCBKGND2=&64
SCALER_DISPGAMADR=&78
SCALER_DISPGAMDAT=&E0

DIM code% 1024
FOR A%=0 TO 2 STEP 2
P%=code%
[OPT A%
SWI "OS_EnterOS"
MOV R0, #13
MOV R1, #IO_BASE
MOV R2, #IO_BASE_SIZE
SWI "OS_Memory"
ADD R3, R3, #SCALER

LDR R0, [R3, #SCALER_DISPCTRL0]
TST R0, #1<<31
LDRNE R1, [R3, #SCALER_DISCBKGND0]
ORRNE R1, R1, #1<<29
STRNE R1, [R3, #SCALER_DISCBKGND0]

LDR R0, [R3, #SCALER_DISPCTRL1]
TST R0, #1<<31
LDRNE R1, [R3, #SCALER_DISCBKGND1]
ORRNE R1, R1, #1<<29
STRNE R1, [R3, #SCALER_DISCBKGND1]

LDR R0, [R3, #SCALER_DISPCTRL2]
TST R0, #1<<31
LDRNE R1, [R3, #SCALER_DISCBKGND2]
ORRNE R1, R1, #1<<29
STRNE R1, [R3, #SCALER_DISCBKGND2]

MOV R1, #1<<31
STR R1, [R3, #SCALER_DISPGAMADR]
MOV R1, #0
.L1
  STRB R1, [R3, #SCALER_DISPGAMDAT]
  ADD R1, R1, #1
  CMP R1, #256*3*3
BNE L1

SWI "OS_LeaveOS"
MOV PC, R14
]
NEXT

CALL code%

dispmanx_offline=1 makes no difference, I’m still getting a blank screen within seconds of reaching the desktop.

I can second that, my RPi3B+ has just blanked itself, it just took a bit longer than the above!

I had a brief look at the Gamma blanking […] It’s either a “feature” of the VC4 or more likely, there’s not enough bandwidth for the HVS to keep the output buffer full, so it shuts down.

The HVS running out of bandwidth didn’t seem to be the case as far as I can tell, I mention it specifically in the ticket because (via VNC) there were no HVS asserts and the only layers turned on were the main desktop and the mouse pointer – both of which have tiny ‘costs’. It was 1280×1024 at 16M and would run for many hours just fine (also noted in the ticket).

The HVS running out of bandwidth didn’t seem to be the case as far as I can tell

We need a PRM for the HVS as we really need to query its state. It certainly looks like its shutting down as the screen signal is still there, there’s just no HVS output to display.

I’ve searched the net and found nothing, so I’m presuming it’s NDA?

Something interesting happened this morning whilst testing, the screen blanked a few seconds after reaching the desktop and just when I was about to reboot, it came back, stayed for a few seconds and then blanked again for good.

If you want to run 5.25 beta, but disable gamma. Place the following in !Boot.Choices.Boot.Tasks:

IO_BASE=&3F000000
IO_BASE_SIZE=&1000000
SCALER=&400000
SCALER_DISPCTRL0=&40
SCALER_DISCBKGND0=&44
SCALER_DISPCTRL1=&50
SCALER_DISCBKGND1=&54
SCALER_DISPCTRL2=&60
SCALER_DISCBKGND2=&64
SCALER_DISPGAMADR=&78
SCALER_DISPGAMDAT=&E0

DIM code% 1024
FOR A%=0 TO 2 STEP 2
P%=code%
[OPT A%
SWI "OS_EnterOS"
MOV R0, #13
MOV R1, #IO_BASE
MOV R2, #IO_BASE_SIZE
SWI "OS_Memory"
ADD R3, R3, #SCALER

LDR R0, [R3, #SCALER_DISPCTRL0]
TST R0, #1<<31
LDRNE R1, [R3, #SCALER_DISCBKGND0]
BICNE R1, R1, #1<<29
STRNE R1, [R3, #SCALER_DISCBKGND0]

LDR R0, [R3, #SCALER_DISPCTRL1]
TST R0, #1<<31
LDRNE R1, [R3, #SCALER_DISCBKGND1]
BICNE R1, R1, #1<<29
STRNE R1, [R3, #SCALER_DISCBKGND1]

LDR R0, [R3, #SCALER_DISPCTRL2]
TST R0, #1<<31
LDRNE R1, [R3, #SCALER_DISCBKGND2]
BICNE R1, R1, #1<<29
STRNE R1, [R3, #SCALER_DISCBKGND2]

SWI "OS_LeaveOS"
MOV PC, R14
]
NEXT

CALL code%

More interesting findings from testing today:

• The program above to disable the gamma may be triggering the issue. With it in Tasks, my screen is blanking 3 out of every 4 boots the second it runs. I had to move it out and run it manually in the end.
• The mouse pointer is possibly affecting the issue. I managed to trigger a blank several times today by moving the pointer down to the bottom right of the screen. I’ve also noticed an increase in screen blanking when the pointer image is updated.

I lost count of the amount of times its occurred today, over fifty times. I had to go back to 5.23 in the end as I was spending more time rebooting than developing. It’s definitely worse if you’re using the mouse on a high-res screen, at various points I was testing mode 28 for the desktop mode and could not get it to blank once.

There is a new cmdline.txt option in CVS :)

There is a new cmdline.txt option in CVS

Working here on a local ROM build. Brilliant, thanks.

We have a response on the github forum

It is not immediately clear to me how to implement the suggested diagnostics but I could probably manage an HDMI hotplug.

I have removed the disable_gamma switch on the RPi3B+ which is so far resolutely not shutting down.

I suppose I should give hotplugging a try as I can reproduce the issue at will!

Hotplugging doesn’t recover the VideoCore. Re-ran the test with music playing in the background, to see if a hotplug gets that working again – it doesn’t.