localtime() is broken [might be Territory?]

The example is simple:

meh = time(NULL);
printf("%s\n", asctime(localtime(&meh)));

If I use gmtime() then the display is 8pm as expected.
However, if I use localtime(), then the display is 9pm.
RISC OS’s *Time informs me that it is, in fact, 10pm.

System configuration is UK territory, daylight savings, and +1h timezone.

While this is technically invalid (there is no UK+DST+1h), given the difficulties of creating territory modules, are we going to cling to some pedantic notion of what is a valid time?
Or can we do international RISC OS users a favour and permit the timezone to be used to modify the base time plus DST (like how RISC OS itself does it)?

I seem to get the same results.

In Configure > Time and date I have Locality set to UTC+12 and *Time returns the correct local time. gmtime() also returns the correct UTC time.

localtime(), on the other hand, returns UTC if DST is disabled, and UTC+1 if it’s enabled. I don’t know how to set/view the Territory (is there a UI for that or do I have to do it from the CLI?) so I’m guessing that mine’s set to UK.

It seems that localtime() always uses your selected territory, which is probably UK on the majority of systems (especially for people like me that have only found RISC OS via the Pi – I didn’t even know that there was such a thing as a “territory” until now).

If the UK is a territory then I’m assuming that a territory is usually a country. In that case I shudder to think what happens if you set your territory to a country that spans multiple time zones!

Ought to be a dead easy fix.

UTC plus DST offset plus time zone offset.

It would appear to be the latter part that we are missing?

Long and short is:

• you’re using UK territory, but with manually adjusted timezone, therefore Territory_ReadCurrentTimeZone returns “unknown timezone for this territory” so the C library is unlikely to get anything local time related right, and automatic DST correction also wont work for the same reason, and the %Z formatted for strftime() that sort of thing
• setlocale() as of 5.20 accepts a named timezone to mimic (for example) Windows, so you can even have one program running in two task windows in two different timezones if you want (!)
• you’ve not called setlocale() however
• there’s even a test program Sprow wrote castle/RiscOS/Sources/Lib/RISC_OSLib/test/c/tzones

The core of the problem appears to be that you’re an English speaker living outside the UK and you’re trying to bodge RISC OS by using the UK territory with a manually set timezone. Territories are about regions of the earth’s surface and they tie up the language and the geography, that *TIME shows the “right” time is a poor measure of what the territory system is trying to achieve.

There is a ‘France’ territory module on the common downloads page (“Territory modules” at the bottom), though I guess you might want to roll a special one with the week/month names in English. Or, for ultimate bodge, add a second timezone within the UK territory and selected that with

  *CONFIGURE CEST

Territories are about regions of the earth’s surface and they tie up the language and the geography

Hmmm? Which of the official languages of Belgium is tied to the timezone they are in?¹

Or should we move a little further inland and ask which of the Swiss languages their timezone is nailed to?

Rick was making the point that: Language <> timezone and timezone <> language.

¹ Four IIRC, Flemish, French, English, and German. The last being a small enclave

I don’t know why the behaviour is as strange as Steve describes.

The sane behaviour would be: If you have Auto Timezone, use the Territory to determine which timezone is the correct one. If the user has gone to the length of manually setting the (no doubt for him correct) Timezone, just trust the user and use it.

Ah, wait. If I have understood all this correctly, I am inclined to say that the currently implemented behaviour is not strange, but broken. Completely broken.

Four IIRC, Flemish, French, English, and German. The last being a small enclave

English an official language in Belgium? Doubt it. Dutch/Flemish, French and German, as far as I know.

Completely broken.

Indeed, totally broken behaviour.

English an official language in Belgium? Doubt it. Dutch/Flemish, French and German, as far as I know.

I bow to greater knowledge, although I think you’d concede that Wallonians would probably rather speak English than Flemish¹ and this is reciprocated in the north. On visits I always used to ask what people would prefer me and friends to attempt to speak: several times the answer was English, you seem quite good at that², but don’t speak German.

¹ Probably rather not speak than not speak French :)
² Quirky humour, unsurprisingly from a native Flemish speaker. A trait I’ve come across from people on the dutch side of the border too

Long and short is:

…Territory is Not Fit For Purpose.

you’re using UK territory, but with manually adjusted timezone,

And? I want my computer to be fundamentally “British”, only I’m not living in the UK. Given that RISC OS provides no alternative…

setlocale() as of 5.20 accepts a named timezone to mimic (for example) Windows, so you can even have one program running in two task windows in two different timezones if you want (!)
you’ve not called setlocale() however

(my highlight)
Why on earth should I? I can appreciate that it is useful to have a function to change timezones; but getting the basic system time is something the system should sort out. It really shouldn’t have to be up to the program to try to figure out where it is because some part of the OS is incompetent.

Furthermore, it looks as if setlocale() uses a named “country/timezone” pairing; so we can’t even try to read the configured timezone offset and say “bodge it by 60 minutes”. This brings in all sorts of additional complications – not only trying to backparse a timezone into some sort of named version, but the necessity – one would presume – to have some sort of territory support for knowing which timezone names are which, and everybody else using said software to have this. Out of interest, what country would you classify the HNA timezone as? :-)

The core of the problem appears to be that you’re an English speaker living outside the UK

The core of the problem appears to be that the people that originally designed the Territory system really had little idea about how to implement such a thing. If they had actually bothered to think a little, they might have realised that their close international neighbours offer countries with multiple simultaneous languages. There is a hint at what a bodge this is in the “Canada (we speak English)” (#17) and “Canada (ici on parle francaise)” (#18) territories, or the fact the using a weird keyboard layout switches you to a different territory (#70 vs #1; ditto US).

and you’re trying to bodge RISC OS by using the UK territory with a manually set timezone.

I’m actually hoping for the supposedly-smart bits of RISC OS to do what the kernel can do effortlessly.

Territories are about regions of the earth’s surface and they tie up the language and the geography,

Fair enough. I bet that looks good on a powerpoint slide. :-P

Problem is – I’ve just pointed out above that Canada has multiple territories to get around the sticky language issue (and I notice you have “Canada1” and “Canada2” instead of “Canada-except-Québec” and “Québec” :-) ), Switzerland with its four languages and Belgium with its two count as only one country each. “Middle East” encompasses French, maybe English, Hebrew, Arabic, and probably some languages I’ve never heard of. Sure, there is an “Israel” Territory, but Israel is in the “Middle East” isn’t it? “UK” and “UK-with-a-Dvorak-keyboard” are different territories. How does that tie in with language and geography? Where does one select “Cornish” as a language? Or Basque? Or Breton? What is the language of “Ireland”? In fact, we have “Wales”, where is “Scotland” (the other country of the UK (Wales is not a country…yet)) and “Northern Ireland”?

Essentially, Territory is Not Fit For Purpose.

that *TIME shows the “right” time is a poor measure of what the territory system is trying to achieve.

That the kernel gets it right and the Territory system does not certainly shows what the Territory system is failing to achieve.

There is a ‘France’ territory module on the common downloads page (“Territory modules” at the bottom),

No good.
Tried it – and the first thing that I see is my network fails to start up (can’t add route). Configuring back? Well, *Config. Territory UK says “(Number)”. The reason is simple…

• If I was to choose a “France” territory, applications that intelligently probe the system will speak to me in French. I do not want this.
• If I set up a “France” territory module, the OS starts to fall apart as it itself starts looking for France resources, and apparently not everything is able to sanely revert to UK messages. I do not want this.

though I guess you might want to roll a special one with the week/month names in English.

I do not want this!

Good god – we’re supposed to roll our own Territory modules just to get the time to show up right in C programs?
Do you understand how unbelievably batpoop-crazy that is?

Furthermore, it is – ultimately – useless – as it implies that CLib is pathologically unable to provide the correct time to anybody that is not running a custom-hacked Territory. I build my own OS (’cos I can, mainly) – how many people here live outside the UK, speak English, and do not build their own OS but just download the latest one from this site?

Or, for ultimate bodge, add a second timezone within the UK territory and selected that with
*CONFIGURE CEST

That would work for me for the summer. ;-) Anybody here using a British-setup RISC OS in a timezone further apart? What about them?

but in both cases leave the timezone configure keyword alone!

I think I’m going to poke around the “UK” territory to see if I can patch in timezone offset bodging. It can reply with the timezone names “WNTR” and “SUMR” (for Winter/Summer time), it will be hardwired to the UK start/end dates, but it will take account of the configured TimeZone offset. This should have no effect whatsoever on UK-in-the-UK systems as your offset will be zero. It will, on the other hand, permit overseas “UK” users to have proper times returned to their programs without the nefarious side effects of using alternative “territories”.
When it is done, I trust that ROOL will give the modification(s) due consideration.

What should I do when I’ll go in mission in Korea?

[edit: it’s also a problem in the C library, but I can still tweak the UK territory for additional timezones in the hope of the C library being fixed]

Okay – it looks like the quickest and cleanest way may be to ditch the use of *Configure Timezone and bake a variety of additional timezones in to the UK territory.¹²

Obviously, CET (UTC +1h00) and CEST (UTC +2h00) will be provided.

Any others to add? Write the official name and the offset from UTC in the box below…

¹ This is still not the best way, any auto-DST stuff will have UK changeover rules, that may work for Europe, but probably not for other places.

² If there are only a few commonly desired options, all is okay. If many, I’ll probably need to go the TimeZone offset route. Which looks like it’d be messy botching of tables. Hmmm…

Well, I have modified the UK Territory to understand CET and CEST.

Doing *Configure CEST sets the DST option and the timezone to +1h00, as expected.
Reading the time in a C program…

…returns BST.

Okay. Something is going wrong here. I have built a standalone version of the USA territory module, and no matter which of the (non-DST) timezones I select (EST, CST, MST, PST, or AKST) – a program using localtime() always returns the EST time. The *Time command returns the expected time. Calling SYS "Territory_ReadCurrentTimeZone" returns the expected information for the timezone in use.

Curiouser and curiouser…

DIM c% 5, b% 255
c%!0 = 3
SYS "OS_Word", 14, c%
SYS &4304B, -1, c%, b%, 256, "%24h%MI.%SE" TO , n%
n%?0 = 13
PRINT $b%

works as expected, for CEST and PDT.

Time to look at what CLib is doing.

Found it.

The localtime() function (source here) is using Territory_ReadTimeZones and making assumptions based upon the [first|only] timezone read and the state of the DST flag.

This can never work for territories with multiple time zones.

Proof:

*Territory USA
*Configure PDT
SYS "Territory_ReadTimeZones", -1 TO std$, sum$, std%, sum%
PRINT std$, sum$, std%, sum%
EST       EDT         -1800000  -1440000
SYS "Territory_ReadCurrentTimeZone" TO zn$, off%
PRINT zn$, off%
PDT         -2520000

The code could be a lot simpler by just asking Territory for the current timezone…
https://www.riscosopen.org/tracker/tickets/391

Actually – this would be an all-round win with no modifications necessary to the UK Territory (for now!).
I set CEST and killed the UK module and reinitialised the one in the ROM image. It doesn’t understand CEST. However asking for the current time zone returned “Custom” for the name, and the correct offset. Therefore, if the C library just asks for the timezone offset properly, then we can set/unset the DST as applicable, and set our own preferred timezone offset manually, and it’ll just work for having localtime() return the correct time.

Finally…

There’s (still) a lot wrong with Territory, however:

[Steve R.] and you’re trying to bodge RISC OS by using the UK territory with a manually set timezone.

and:

[me] Good god – we’re supposed to roll our own Territory modules just to get the time to show up right in C programs?
Do you understand how unbelievably batpoop-crazy that is?

It looks like Territory can actually handle this situation and it’s a library problem¹. So on the point of custom timezones with the UK territory, I will apologise to TerritoryManager. Sorry. Sumimasen. Désolé.

¹ Just looked at the earliest source available, some 17 years ago, and it did it like that back then, too. Perhaps, one could wonder, it did it this way since forever.
I can’t help but think that it probably isn’t a good thing that nobody noticed until now… Erk!

It’s late, hot, hard day at the office, blah blah. I’m going to microwave a quiche, eat it, then go to bed. Night guys!

I’m still getting my head around all this, but it looks like multiple territories for Canada, Belgium, etc. is the “correct” way. As per PRM 3-797:

Sometimes you may need to provide more than one territory for a single country. For example, to properly support the whole of Switzerland you would need a separate territory for each for the languages used.

But as you noted, that doesn’t make localtime() work correctly!

Damned. So you should add french (keyboard, language, etc.) to USA, all Europe and part of Asia, since I go very often in these parts of the world…

IMHO, it would be more simple to separate timezones from territories.

IMHO, it would be more simple to separate timezones from territories.

Or, when the C library is fixed, just set the timezone manually…

Will be submitted “officially”, however to those who want to try it/play with it now:

1. How to add extra timezones to the UK territory

This is not necessary, Territory will work with custom timezones. This is so you can do *Configure CEST instead of *Configure DST followed by *Configure TimeZone +1 so it is up to you if you bother.

Open the UK Territory source – …castle.RiscOS.Sources.Internat.Territory.Module.s.UK – near to the top you will see:

MaxTZLength     *       3       ; "GMT"

Alter this to suit the timezones you are adding – “CEST” would be ‘4’.

Just below this, you will see:

NumberOfTZ      *       1

Alter this according to the number of timezones you are adding. DST/NoDST counts as one timezone, so to add CET/CEST, we’d set this to be two.

You will then see this:

        GBLS    NODST0
NODST0  SETS    "GMT"

        GBLS    DST0
DST0    SETS    "BST"

NODSTOffset0    *       100*60*60*0     ; Zero hours
DSTOffset0      *       100*60*60*1

What we need to do is basically copy this, change all the zeroes to ones, and then fill in the correct information.

To add CET/CEST, we would add this following the block shown above:

        GBLS    NODST1
NODST1  SETS    "CET"

        GBLS    DST1
DST1    SETS    "CEST"

NODSTOffset1    *       100*60*60*1     ; NoDST "CET"  is UTC +1h00
DSTOffset1      *       100*60*60*2     ; DST   "CEST" is UTC +2h00

A subsequent timezone would be NODST2, DST2, NODSTOffset2, and DSTOffset2; and so on.

Save, rebuild your ROM, you’re all set.

Tip! If you want to build a non-ROM version of the module for testing that it works, go to the parent directly and you’ll see a load of taskobey files called !Mk<something>. These are the commands to make the relevant territory modules.
Copy the one called “!MkUK” as “!MkUKSA”, then load “!MkUKSA” into an editor.
You’ll see at the bottom it says:

amu_machine rom COMPONENT=UK

Change this to say:

amu_machine standalone COMPONENT=UK

and then you just need to load the !Builder application (sets paths and stuff) and then double-click on “!MkUKSA” to build a standalone version of the UK territory. [same applies for the other territories]
You will find your module in rm.<cputype> – so for the Pi, it’ll be rm.ARM11ZF.UKSA. If it doesn’t have the “SA” suffix, it is probably a ROM build – you did run the right build file? ;-)

And now for the important modification:

2. Fixing localtime() to work correctly

The source for the localtime function is at the end of this file – castle.RiscOS.Sources.Lib.RISC_OSLib.c.time – and rather than provide bits to change, I’m just going to list the entire function. Copy-paste this in instead of the existent version…

struct tm *localtime(const time_t *timer)
{
    time_t t;
    static struct tm _tms;
    int dst;
    int territory;
    int v;
    _kernel_swi_regs r;

    t = *timer;
    if (t == (time_t)-1) {
        memset(&_tms, 0, sizeof(_tms));
        _tms.tm_mday = 1; /* 1st day of 1900 */
        return &_tms;
    }

    // Read DST flag (it is needed for time_to_tm at the end in any case)
    dst = -1;
    v = _kernel_osbyte(161, 220 /* AlarmAndTimeCMOS */, 0); // read CMOS
    if (v >= 0)
       dst = v & 0x8000; // DST flag is bit 7

    // What locale are we using?
    territory = __locales[N_LC_TIME];

    if ( !territory )
    {
       // If C locale, use the currently configured timezone
       // of the currently configured territory.
       r.r[2] = 0; // make sure it does not say "ZONE"
       _kernel_swi(Territory_ReadCurrentTimeZone, &r, &r);
       t += (r.r[1] / 100); // centiseconds -> seconds
    }
    else
    {
       // Custom locale - work it out from the specified timezone and DST setting

       // read timezone information from Territory
       r.r[0] = TERRITORY_EXTRACT(territory);
       r.r[1] = TERRITORY_TZ_EXTRACT(territory);
       r.r[4] = TERRITORY_TZ_API_EXT; /* If not supported, never mind */
       if (_kernel_swi(Territory_ReadTimeZones, &r, &r) == NULL)
       {
          v = (dst == 0x8000) ? r.r[3] : r.r[2];
          t += v / 100; /* centiseconds -> seconds */
       }
    }

    /* Already corrected for locale, so just need to mangle it */
    /* into a suitable structure                               */
    return time_to_tm(&_tms, t, dst);
}

The previous behaviour has been retained for specific locales. This has not been tested, but I’m assuming it works and was tested as part of the development of the specific-locales code, for stuff like passing “USA/PST” to setlocale().

For our needs, if we are running in the default locale (which you will be unless you specify otherwise), we ignore the DST flag (it is passed to time_to_tm() so the user will know if we’re in DST or not). Instead, we just ask the territory system for the current timezone offset. We don’t correct this value, Territory has already done this for us.

Tested with UK: GMT, BST, CET, CEST.
Tested with USA: EST, EDT, PST, AKST.

In all cases, the time returned by asctime(localtime(&meh)) matches *Time.

Thunderstorm approaching, time to unplug the internet. Don’t want to lose another Livebox…

A couple of questions:

1. Is there any documentation on how to create a new territory module, or should I just duplicate one of the existing ones and modify it in whatever way seems sensible?
2. The source files refer to Maori as territory 37, but it’s not listed on the Territory Numbers page. Should it be?

Documentation – PRMs? If not, yeah, I’d just clone one of the existing modules.

But note that there is a massive lurking “gotcha!”. Unless you are adding a new language, it would be bad to clone a module to try to add different behaviour to an English language system; because just about everything assumes that the territory name indicates the language.
This is why I hacked UK. It would be no good to make a new territory called “English”, because apps keep English in a “UK” resource file, determined from the territory name (there is no “English” resource, and apparently no way to tell everything to “just use the UK files”.

Time for me to duck out for today. Walking around the field in between the rain and thunder clouds to try to get a good mobile signal is kind of silly.
Here is “present time, present day”: http://i.imgur.com/B3gbjrf.jpg

Documentation – PRMs? If not, yeah, I’d just clone one of the existing modules.

I couldn’t find anything in the PRMs but admittedly I didn’t have a thorough look.

Unless you are adding a new language, it would be bad to clone a module to try to add different behaviour to an English language system; because just about everything assumes that the territory name indicates the language.

Time for me to play around with the USA and Australia territories and see what happens. Presumably the system still works when they’re selected!

I have put together some ideas for ways we could enhance the territory/language system. You have “comment” access, so don’t be afraid to annotate. ;-)

https://docs.google.com/document/d/15RwrmIzUF8r9iJOXX2BEXeMvJkYYSut88yJNaE1cozQ/edit?usp=sharing

Doesn’t need you to sign in to annotate (just tried with iOS/Dolphin), so please append your forum name if you aren’t signed in, m’kay?

Thanks.

Following my earlier comment you will appreciate that I totally agree that language and territory or country are not synonymous.

One small amendment: Belgium has three official languages (I looked it up following the response from Frank)

Note: English is widely spoken but not official.