localtime() is broken [might be Territory?]

I agree too :)
Territories link things that should not be. Especially on RISC OS that is very flexible.
(Sorry to be direct).

I have put together some ideas for ways we could enhance the territory/language system.

There is an element of deja vu here, but this is slanted towards time rather than languages/numbers/currencies. I agree that the Territory concept seems to fall short of what is required. However, I seem to recall that it was first conceived as a Rom module for machines that would be sold both inside the UK and overseas, each of which would be a separate version. It would provide reasonable defaults for a system out of the box. This has had onward implications.

After the last flurry I wrote a suite of ‘Region’ programs that allowed independent setting and configuration of most of the factors that were of concern. I do not know Windows 7 or 8, but a good look at the ‘Regional and Language’ dialogue in XP suggests that its progammers might have come up against some similar problems.

I never put the final polish on the suite, for consideration here, for two main reasons. First, the Configuration plug-in and it sub-programs are written using Basalt. Second, I used soft-loaded Territory modules that were modified in situ. Nevertheless, the whole thing was great learning experience. ;-)

The modified module method was quite simple, because you can identify the various tables from the values returned by SWIs. I have done a quick test, and it is certainly possible to change the default time zones and names this way. So you could configure the France Territory on the fly with GMT amd +0 instead of CET and +1. This is more complex than that simple statement, and I do not think modifying modules is likely to be an approved technique, but I offer it as a possibility, because it does not need a whole new system and should be backward compatible.

There is an element of deja vu here,

Yup. ;-)

but this is slanted towards time rather than languages/numbers/currencies.

That part gets the largest paragraph in the document I linked to, mind you. ;-)

I agree that the Territory concept seems to fall short of what is required. However, I seem to recall that it was first conceived as a Rom module for machines that would be sold both inside the UK and overseas, each of which would be a separate version. It would provide reasonable defaults for a system out of the box.

Which no doubt made some sort of sense in 1992 (although I suspect the two Canadas, the one Switzerland, and the Dvorak keyboard as a new territory should have hinted that the spec was going to fall apart sooner or later).

I do not know Windows 7 or 8, but a good look at the ‘Regional and Language’ dialogue in XP suggests that its progammers might have come up against some similar problems.

It is “something like that” that I would have in mind for the user-facing interface.

I never put the final polish on the suite, for consideration here, for two main reasons. First, the Configuration plug-in and it sub-programs are written using Basalt.

? Aren’t you supposed to be promoting Basalt? ;-)

Second, I used soft-loaded Territory modules that were modified in situ.

Ah. Well… Doesn’t the necessity to load a soft-load module and binary-patch it in-situ suggest something of a grand limitation of the original design? Anything that needs to be poked like that is either a game the user wants to cheat at, or badly designed.¹

I have done a quick test, and it is certainly possible to change the default time zones and names this way. So you could configure the France Territory on the fly with GMT amd +0 instead of CET and +1.

Doing it the other way around? Less so. The UK module holds the timezone in four bytes as both are three characters long. Pasting in CEST would require the MSB of whatever follows to be a null to act as a string terminator. That’s icky.

Certainly, adding additional time zones would be non-trivial. I guess you could allocate some RMA, build the table there, then fudge the pointers in the module…but that’s about the time to concede that there ought to be a better way!

it does not need a whole new system

I don’t really fancy designing and creating a whole new system – but the more I try to do non-UK, the more Territory gets in the way.

and should be backward compatible.

This is a big concession that I made. A lot of stuff (all of the ResFind type programs, for instance) make the assumption that territory name equals language. This is why we all stick our resources in “UK”. As awful as this may be for any sensible attempt at internationalising things, it’s what we are stuck with at the moment, and I fully acknowledge that my idea (and its implementation) might be the best thing ever, me on the front of Time magazine, etc – but it won’t be worth a damn if it breaks everything written prior (which, let’s face it, will be everything). As much attention needs to be paid to faking something for compatibility as making something new. There’s no if, but, or maybe about that.
[though, your nice front end will cease to work, as there wouldn’t be a UK (etc) module to poke and prod; never mind, the pokey-proddy API should be easy to use (^_^)]

¹ Or we’re doing nefarious things with copy protection, the less said about that…..

A lot of stuff (all of the ResFind type programs, for instance) make the assumption that territory name equals language.

Actually, the original ResFind used the “Country” as the default, not the territory name. And, using ResConf, the user had an easy way to choose per-app language. Which was an excellent reason for many developers to choose ResFind as an i18n vehicle instead of inspecting the territory.

Time for me to play around with the USA and Australia territories and see what happens. Presumably the system still works when they’re selected!

Well, I made a new territory cloned from Australia, and everything seems to work properly. *Time gives the correct time for both timezones, and localtime() as expected gives the correct time for the first one in the list (I haven’t applied Rick’s fix). A quick check of a few apps didn’t turn up any problems; the apps appeared in English as expected, presumably using UK resources.

The only problem I had was that I had to “piggyback” on top of an existing territory because I didn’t know how to add a new one; editing RiscOS.Sources.Programmer.HrdSrc.hrd.Countries doesn’t appear to be sufficient. Ideally I’d ask for one to be allocated properly but I’m just trying to test things at this stage.

PS. I just realised that I haven’t mentioned it yet: I’m in New Zealand and there is no existing territory number. Bizarrely there’s one for Maori (NZ’s other official language) so I hijacked that one for testing.

It is “something like that” [the ‘Regional and Language’ dialogue in XP] that I would have in mind for the user-facing interface.

It was a guide for me, but it cannot be copied. First, we have to fit into the RO Configure protocols. Second, Windows gives primacy to the language, so there are choices for German (Switzerland), French (Switzerland), Italian (Switzerland) and Romansh (Switzerland) rather than Swiss1, Swiss2, Swiss3 and Swiss4.

Aren’t you supposed to be promoting Basalt? ;-)

I would, but it is a third-party resource and will not be accepted for system components. ;-(

Doesn’t the necessity to load a soft-load module and binary-patch it in-situ suggest something of a grand limitation of the original design?

Yes, but bear in mind that the module being patched is subsidiary to the Territory module and that it is largely a data repository.

Pasting in CEST would require the MSB of whatever follows to be a null to act as a string terminator. That’s icky.

I was very careful to choose and example that worked, so you caught me out there. If this were a reasonable method to adopt, it would be quiet simple to change the module format so that strings were in , say, 12-byte buffers.

A lot of stuff (all of the ResFind type programs, for instance) make the assumption that territory name equals language.

I have no idea how this can be resolved in retrospect. Like you, I can see a way to design things for the future – my Obey$Path macro-variable works for me – but I have only the haziest ideas about making that work with ‘UK’ etc.

the original ResFind used the “Country” as the default, not the territory name

Although you can make them different, by default they are the same, and share the same list. I understand that Terrirtory should normally take precedence.

It’s a little hard to pick out the content here amongst the angry mob shouting, a more calm and considered approach is generally what’s needed in engineering disciplines.

setlocale() as of 5.20 accepts a named timezone to mimic (for example) Windows so you can even have one program running in two task windows in two different timezones if you want (!), you’ve not called setlocale() however

Why on earth should I?

A case in point: the two statements are correct – you can pass a named timezone to setlocale(), but in the original post it looks like setlocale() is not being called, if setlocale() isn’t called you get whatever the C library chose to implement.

UK and UK-with-a-Dvorak-keyboard are different territories. How does that tie in with language and geography?

I think there’s some confusion here because countries, keyboards, and alphabets are all (currently) allocated from the same creaking number space. As it happens countries map 1:1 to territories for those < 127 but the Dvorak allocations are keyboards. However, as there’s no configure command to store the keyboard it’s (currently) spoofed as a fake country. Some keyboards have leaked into the wiki too I see.

The localtime() function is using Territory_ReadTimeZones and making assumptions based upon the [first|only] timezone read and the state of the DST flag.

Looking back in CVS history it appears to have always done that. However, when Territory_ReadTimeZones was fixed to not adjust the lookup table based on timezone (which never really made sense, since things like CET and CEST are at constant offsets from UTC) it appears to have exposed a latent bug in localtime(). A bug in software? Bring me my flaming torch…

With the C library extra care is needed

• is the change ISO9899 compliant?
• what happens if I softload the change on a pre RISC OS 5 machine?

I don’t see any evidence of this in ticket 391.

I’m still getting my head around all this, but it looks like multiple territories for Canada, Belgium, etc. is the correct way. As per PRM 3-797

The allocation of territory and country numbers is likely to be incomplete, for example if a flemish speaking commercial customer from Belgium commissioned Acorn to produce a translation of RISC OS for them why would they pay for all the work of a French and German translation too? Therefore, an absence of an allocation doesn’t mean such a territory couldn’t exist.

Canada1 and Canada2 are probably the cleanest examples of what was intended, though I don’t think that infers needing two modules as that PRM reference suggests. Rolling a one module = one territory isn’t hard, the entry points are mostly just pointers to tables, and there’s plenty of examples to start from.

There is an element of deja vu here, but this is slanted towards time rather than languages/numbers/currencies.

Agree – this has swerved away from a bug and more into design. However, in reading I don’t really see anything that wasn’t already captured in this thread which in itself repeats much of this thread so if there’s actually some new ideas it’s probably worth pasting them onto the end of one of those 2 threads so they don’t get lost.

amongst the angry mob shouting

Wow – about the only time Burns and Homer have been in agreement over something.

A case in point: the two statements are correct – you can pass a named timezone to setlocale(), but in the original post it looks like setlocale() is not being called, if setlocale() isn’t called you get whatever the C library chose to implement.

So you are saying that it is the responsibility of the application to handle timezone offsets because the C library is broken ? Note that setlocale() doesn’t accept a numerical offset, so the application needs to be able to translate time offsets into named zones, then enumerate the available territories….then probably fail as this depends on a territory with this timezone being present, and possibly fail because the C library is broken.

is the change ISO9899 compliant?

Quote:

7.23.3.4 [The localtime function]

 1          #include <time.h>
           struct tm *localtime(const time_t *timer);
    Description
 2   The localtime function converts the calendar time pointed to by timer into a
    broken-down time, expressed as local time.
    Returns
 3   The localtime function returns a pointer to the broken-down time, or a null pointer if
    the specified time cannot be converted to local time.

[source]

Note the repeated references to local time. I would imagine an implementation that supports time zones but has a localtime() erroneously hard wired to return always only the first timezone pairing would itself be non-compliant; if you are in the zone PST it returns EST time (not null), thus the result is incorrect.

what happens if I softload the change on a pre RISC OS 5 machine?

Nothing. I say this because there is no indication in the CVS that it ever did it another way, so I can only imagine that this goes back to RISC OS 3.00 and in the intervening decades, apparently nobody else noticed. Wow. Of course, for completeness, I ought to write a small test program and try it on various emulated versions of RISC OS…

Oh, and the SWI used is documented in the RISC OS 3 PRM, so it should be valid on everything that isn’t RISC OS 2.xx.

Therefore, an absence of an allocation doesn’t mean such a territory couldn’t exist.

Of course, that’s what the allocations are for. However, as pointed out, there is the assumption that the territory name implies the language, which really ought to be “aided by the territory but apart from it”. This would allow combinations like UK/Welsh and France/Breton. Things that are not possible now without creating entirely new territories. Think also of Morocco and Lebanon, both could validly use French as a language. Or how many countries could use English as a language, and have their own regional options. The entire commonwealth for a start. Probably America too (I will worry about localiszed spelling when Microsoft does!).

Timezone….that ought to be a separate module that just deals with time zones independently of any specific loaded territory.

Or how many countries could use English as a language, and have their own regional options.

Wrapping in timezone too, I’d offer UK, Australia, New Zealand, and the US as a prime example of spread of primary language use in disparate locations/regions/timezones.
Of course you could choose Spanish or Portuguese and have a different spread.

The different territory, timezone, language and keyboard use do need dividing.

The question is what is the least disruptive change method?

I will worry about localiszed spelling when Microsoft does!

Perhaps this isn’t the best time for me to point out that Windows 8 is available in “real” English :)

@ Chris: So with their budget and market dominance, it only took ‘em 20 some years to recognise that Brits spell stuff differently. ;-)
I played with a laptop running Windows 8 in a supermarket a few weeks ago. What the hell? Does it even multitask ? It seemed to behave more and more like a tablet with a "you’re doing this and this alone" approach. And where did they hide the filesystem? What a mess!

I don’t deny that it’s terrible :)

Still, Microsoft is doing better than Apple; only iTunes and the “iDevices” have been localised.

The question is what is the least disruptive change method?

Best solution is to separate all these parameters, because they are not linked.

• time/date/numbers format > country of birth
• language > country or user choice
• date of hour changes (winter / summer) > country where you are
• timezone > country where you are
• keyboard > country or user choice

Perhaps it’s possible to have a configure plugin that can assemble a territory module from all these parameters? So we keep the module, but give more flexibility to the users.

For example, here in France:

• A classic user needs french keyboard layout, french language and formats, french timezone.
• A UK user needs french keyboard layout, english language and formats, french timezone.
• A RPC user needs UK keyboard layout, french language and formats, french timezone.
• A software developer needs french keyboard layout, english formats, french language, french timezone.
• A guys that travels in US will need to keep keyboard, formats and languages, but to change timezone and date of split for summer/winter time.

None of these parameters can be linked together.

Perhaps it’s possible to have a configure plugin that can assemble a territory module from all these parameters? So we keep the module, but give more flexibility to the users.

I pretty well agree with your list of paramenters and I have written the substance of such a plug-in. However, there are some things to add, there are currently limitations and it is not yet presentable.

Would anyone would use it? Would it be acceptable to fiddle with territories this way? Could third-party software be part of it? Should I work on this further rather than than just for fun? ;-)

@Rick

I have been re-reading about territories in the RO3 PRM. I did note that there are several mentions of the CLib functions and their relationship to Territory SWIs. In each case it states that setlocale() must be called before using any of them. Is this relevant?

Would it be acceptable to fiddle with territories this way?

I read David’s suggestion of the Configure plugin creating a personalised territory module and thought that perhaps a multinational that picked specific settings information from a text file might be the way to go.

Could third-party software be part of it?

I’m sure developers are free to offer their work to be included in the general repository.

Should I work on this further rather than than just for fun? ;-)

Since there’s no real money in these things, fun or sense of achievement is all there is. I say “Play away”.

Would anyone would use it?

Me for sure.

Would it be acceptable to fiddle with territories this way?

To be honnest, I don’t think so. But, evolution is better than revolution, so it seems to be the best way (IMHO).

Should I work on this further rather than than just for fun?

Oh yes :)

I read David’s suggestion of the Configure plugin creating a personalised territory module and thought that perhaps a multinational that picked specific settings information from a text file might be the way to go.

A bit as MDF files. To provide predefined templates for example. Why not. Some 32bit tools to create keyboard and resources modules should be provide by default too (I’m not sure they are still available).

As tools to make pointers ans system sounds modules. A bit out of topic.

In each case it states that setlocale() must be called before using any of them. Is this relevant?

If you don’t explicitly call setlocale(), it is set to the “C” locale, which is the system’s current territory/settings.

“During program startup, the equivalent of setlocale(LC_ALL, “C”); is executed before any user code is run."

A case in point: the two statements are correct […] setlocale()

So you are saying that it is the responsibility of the application to handle timezone offsets because the C library is broken?

No, the two statements were pointing out a shock new feature in the OS that relate to localtime() that you might have been interested in. Clearly you’re not interested, I’ll stop wasting key presses.

* is the change ISO9899 compliant?

7.23.3.4 [snip]

Good, it seems to be compliant. Remember there’s an equally large angry mob that chases after you for fixing bugs in the C library so it’s worth checking.

* what happens if I softload the change on a pre RISC OS 5 machine?

Nothing. I say this because there is no indication in the CVS that it ever did it another way.

I think you’re only looking at the C library then, remember there are 4 combinations to test as the territory module has also changed so by my reckoning the current C library on an older territory (where Territory_ReadTimeZones changed its answer depending on the timezone CMOS) would still mask the localtime() problem.

Unfortunately CVS only goes back as far as about 1996 so an empirical test will be best to cover back to 3.10 (which is the earliest version the C library is offered for).

Just looking at the bug listing and…

Suggest old and new C library on 3.10 3.50 3.60 3.70 4.0x with whatever the ROM vintage territory manager and UK territory is.

I am guessing that I can build a 26 bit version of the C library by running the “MkLibs26” file?

All that this will (hopefully) show is that the change does not screw up the time on older machines. It won’t show up whether or not this issue even exists as for that we would need to have a 26 bit version of a territory with multiple timezones.
Would making a version of !MkUK which says this do the trick?

echo amu_machine standalone SCL_APCS="-APCS 3/26bit" COMPONENT=UK

(my UK territory module has been modified to include CET/CEST)

If so, I can try the result on RISC OS 3.70 (RedSquirrel) and… I might have to dig out “actual hardware” to test on RISC OS 3.10.

Err – RSq runs 3.10 & 5.19/5.21 don’t know about RO6.

RSq runs 3.10

Oh, that’s good to know. I’ll have to see if I have some ROM images for that OS. Can’t now as I’m ripping a DVD for mom set in her home town, and that computer is using the keyboard and mouse that I’d normally use with the Pi.

Okay.

Can’t try RISC OS 3.10. I have a ROM image extracted from !Archie (that works!) but when used with RedSquirrel’s A5000 mode (either as a 2MiB file or split into 4×512KiB) it causes RedSquirrel to crash.

I have built a 26 bit version of my version of the CLib localtime() function. I have been sent some tweaks, but those are on my phone, not anything I can easily get over to RISC OS. Comparing the two, on screen, there is nothing that should alter the code behaviour in any significant way for these tests.
The “PlingSystem” build (using 1st Nov 2014 vintage code) failed with many errors, but at least CLib was built; so I am using that alone.
The boot process in interrupted by hammering Esc and then when that is done, either the test CLib or the original replacement CLib is loaded, then the test program.

The test program is as follows:

#include <stdio.h>
#include <time.h>

int main(void)
{
   time_t now;

   time(&now);
   printf("gmtime() says    : %s", asctime(gmtime(&now)));
   printf("localtime() says : %s", asctime(localtime(&now)));

   return 0;
}

The results are:

RedSquirrel – RISC OS 3.70 – UK territory, GMT timezone

With standard 32 bit neutral CLib:

*timefiddle
gmtime() says    : Mon Feb 16 18:01:17 2015
localtime() says : Mon Feb 16 18:01:17 2015

With modified CLib:

*timefiddle
gmtime() says    : Mon Feb 16 18:11:43 2015
localtime() says : Mon Feb 16 18:11:43 2015

RedSquirrel – RISC OS 3.50 – UK territory, GMT timezone

With standard 32 bit neutral CLib:

*timefiddle
gmtime() says    : Mon Feb 16 18:15:41 2015
localtime() says : Mon Feb 16 18:15:41 2015

With modified CLib:

*timefiddle
gmtime() says    : Mon Feb 16 18:17:32 2015
localtime() says : Mon Feb 16 18:24:45 2015

Yes – the times appear to differ by 433 seconds (7m13s). This appears to be consistent and does not occur with RISC OS 3.70. I do not have RISC OS 3.60 ROM images to compare.

RISC OS 3.10 – UK / GMT

Okay, finally got something running on RISC OS 3.10 (and it was painful, I’m using the !Archie emulator within an emulator – yikes!) and it is also exhibiting the 7m13s disparity; behaviour is as for the RISC OS 3.50 test.
Any ideas? What on earth could skew the results by 7m31s? That’s 433 seconds, or 43300 centiseconds. Sounds like a “read the wrong register” kind of deal, but no, the UTC offset is in R1 (PRM 3-809) and I’m using R1.

Now, to understand why I bother, here are the tests with RISC OS 5.21 using a modified UK territory and the CET (2nd) timezone. However, for the purposes of argument, the behaviour is identical for any territory with multiple timezones where the user has selected a timezone that is not the first one. Consider USA with Pacific time.

RISC OS 5.21, UK territory, CET timezone

Using last night’s RISC OS 5.21 territory and the CET (+1) timezone:

gmtime() says    : Mon Feb 16 19:20:25 2015
localtime() says : Mon Feb 16 19:20:25 2015

This is not correct. There is no way timezone +1 can match GMT. This occurs because the standard localtime() code looks at the first timezone defined and, well, pretty much ignores the rest (including which one is actually in use).

The same test, using my older RISC OS 5.21, with patched localtime() function:

gmtime() says    : Mon Feb 16 19:36:45 2015
localtime() says : Mon Feb 16 20:36:45 2015

This is correct. It is asking RISC OS for the details of the current timezone offset, instead of making assumptions.

Summary:

Something is obviously acting in a peculiar manner to create the minor (7m13s) disparity on early versions of RISC OS. This isn’t going to be an emulator bug as it happens in RedSquirrel and Archie alike; while it works exactly as expected within RISC OS 3.70.
Just had a look at “Blacklog” for CLib and Territory from February 1994 onwards (date of RO3.5), it doesn’t cover as far as July 1996 (date of RO3.7) but nothing the looks like it would account for a change. Then looked at CVS log and, same. Nothing stands out, other than a big 12 year gap → https://www.riscosopen.org/viewer/view/castle/RiscOS/Sources/Internat/Territory/Manager/s/SWIs#rev4.6 For some things, like the “time.c” file, there are fairly concise descriptions of changes in Blacklog and then again in CVS as of RISC OS 5. The versions for 3.60 and 3.70 (etc) appear to be “Imported from blahblah” so were details of changes not maintained, or has this information been lost? Between RISC OS 3.5 and RISC OS 3.7, something has changed. But what? https://www.riscosopen.org/viewer/view/castle/RiscOS/Sources/Lib/RISC_OSLib/c/time.diff?r1=4.1.7.1;r2=4.1;f=h Nuff’ said. :-(

This leaves is with a choice. Okay, it will require more testing on more platforms, however…

• Option 1 – we do not accept a (currently) small (7 minute) disparity between gmtime() and localtime() on obsolete systems and we choose to maintain accuracy despite the fact that the timezone enumeration stuff is broken on said obsolete systems and the library is broken on everything.

or:

• Option 2 – we accept that there is a small disparity, but looking forward and potentially in the future dealing with people from different countries using different timezones. RISC OS is currently rather anglocentric but this may have to change, and if it does – is it really acceptable for an American user to have the correct time set up and his C library to be up to six hours out?
  The C standard says only “The localtime function returns a pointer to the broken-down time, or a null pointer if
  the specified time cannot be converted to local time.”. As we’re now on C11, the older C99 standard can be obtained here → http://www.open-std.org/jtc1/sc22/WG14/www/docs/n1256.pdf

That said, there is an option 3 that may be the preferred option:

• Build time switch. Old systems (that never noticed) can get old code. New systems (32 bit, that is) can get code that works.

Well, that’s what I did on a rainy Monday. ;-)
[bug tracker updated]

PS: Textile sucks. I might have mentioned that before…