c and sqlite
Pages: 1 2
Michael Grunditz (8594) 259 posts |
I am trying to use sqlite with c (norcroft).
“testdb” is in current directory and can be opened with the commandline sqlite. I build it like this.
|
Michael Grunditz (8594) 259 posts |
Postmortem requested The crash. |
Chris Gransden (337) 1207 posts |
I get the same crash. The original 3.3.6 version doesn’t crash. |
Michael Grunditz (8594) 259 posts |
Thanks! Made my day. |
Paolo Fabio Zaino (28) 1882 posts |
Yeah, I think it has been a while that doesn’t build correctly with DDE anymore. However, !sqlite3 Library (IIRC installable via !Packman) builds fine with GCC 4.7.4, no crashes when running the code. I have stopped using it because, when used with a lot of data, just hangs the desktop on long searches etc. |
Michael Grunditz (8594) 259 posts |
Yes Thats the fun with our “multitasking”!! |
Steve Pampling (1551) 8170 posts |
If only the hardware had another core or two that such processes could run in |
André Timmermans (100) 655 posts |
It’s not an hardware problem, we have machines with multiple cores. |
Chris Gransden (337) 1207 posts |
To see it in action copy a large file in a TaskWindow. *copy largefile largefile2 Everything stops until the copy has completed. |
Paolo Fabio Zaino (28) 1882 posts |
Thanks André and Chris. Indeed all true. To make !Launchpad so “multi-tasking” (included a multi-tasking bootstrap) required more code and a specific architecture. For the untrained reader: To give the illusion of multi-tasking on RISC OS desktop is NOT an easy task for all circumstances. Threads and cores don’t help, so please stop mentioning these when it comes down to the Desktop hanging. The problem is not just the I/O (I?O issues have been described well by André and the great example that everyone can try provided by Chris above), the problem is also the WIMP events queue is not designed for multi-core or is even remotely multi-threaded, on top of that there are other parts of the kernel and the WIMP that will need reentrancy, thread-safeness etc. So, the “easy” way to make a piece of code to provide an illusion of multi-tasking on RISC OS Desktop is to WRITE that code expressly for a coopertive “scheduler”. This means that our own program (itself) has to figure out WHEN it’s time to give back control to the OS. This has to happen for EVERYTHING a program is doing. In simple terms this technique is called “Slicing” (and probably you cannot google for it, given cooperative schedulers are so out of fashion that not even google has info about them). In a preemptive scheduler, it’s the Kernel (via an IRQ – Interrupt Request) that decides when to “slice up” our code and take back control. While (again) in a cooperative scheduler, it’s our own code that needs to figure out when it’s the case to give back control to the Kernel (otherwise the unavoidable hang or stop effect will happen and it will, threads or not threads, cores or not cores). For how to determine “when it’s time” to give back control, there are old computing theories and maths (which I’ll show on a video series dedicated to this). When we port modern programs to RISC OS, however, pretty much ALL of modern programs and libraires have the assumption that the OS they will run on has a preemptive scheduler (or, at least what is known as a semi-preemptive scheduler, which is an hybrid form the preempt in a bunch of scenarios and cooperate in others, Linux Kernel had such a scheduler for quite few years and so did older version of Windows). In short, when making a search (for example) over a large1 amount of data, SQLite will keep seeking through index keys, until an index key correspond to what the user wanted to search or it’ll make no more sense to continue searching2. Now, if that search takes just few ms (milliseconds), the user won’t notice the “stop”, but if it takes more, then the stop will become visible (and obviously, the longer it takes the more visible is the stop). To avoid that, there are a number of techniques, of which the most common are: 1) Intercept certain calls an algorithm may do and use those to quickly return to the OS None of this is natively present in SQLite (or any other ported code to RO), so the developer that does the port has to add such slicing up techniques and ensure that everything still works fine. For the second type, it also changes the ported code API and makes it difficult to maintain (which is another problem on top of the more common porting issues). Things gets more complicated for libraries that may be used in a single task environment as well, but you get the idea. Hopefuly this clarifies once and for all the situation. 1 Large is a relative quantity that depends on a set of factors, for instance with SQLite large doesn’t mean huge, because RISC OS Filesystem doesn’t even caches files and so even a moderately big DB can cause Desktop temporary freezes when operating it. 2 This is just a simplistic language contraction to avoid having to describe databasing search algorithms in detail, it would be out of scope. 3 Normally this technique can be used on RISC OS only for code executed during a NULL event. To use it for other types of event it also requires our program to implement a “received events” priority queue. |
Rick Murray (539) 13840 posts |
Did you actually look before posting that? ;)
Technically, “slicing” is the term used in preemptive systems as you’re given a slice of time (and when that time is up, your task is suspended and another one run). The methodology comes from the old time sharing mainframes. When slicing would make sense in cooperative would be when attempting to fake threads by sharing your single task runtime amongst a number of different bits of code.
Or, in our case, the window manager, since the kernel doesn’t know about “tasks”.
Exactly this. When running preemptively, the task can just “do its thing” and the OS will periodically interrupt it in order to get other stuff done. This happens entirety transparently to the task. When running cooperatively, the task must explicitly yield to the OS every so often so that it can go and do something else. Or, to look at it from the other side, in most other operating systems it is the OS that tells the task what to do and when. Both methods have their advantages and disadvantages, but with the greater control allowed by cooperative multitasking comes the greater responsibility of yielding at a suitably fast rate to keep the illusion of multitasking going. Unfortunately, under RISC OS, this is often broken. For example even on today’s hardware ChangeFSI takes a couple of seconds to decode a photo. And for those seconds, nothing happens, the machine pauses. And, of course, run any program written with the expectation of a preemptive system and the machine will pause as even if it could yield, it won’t, because it hasn’t been told to do so. There is an option 3, write a simple preemption system and use that. Of course, it can get complicated. What do you do if the event returned by polling is not a null? |
Rick Murray (539) 13840 posts |
Because a file copy is a single atomic 1 operation. FilerAction cheats by breaking files into chunks and copying bits at a time. Elsewhere in the world, a task waiting for data from a file would be quietly suspended until the file was ready, and likewise periodically interrupted as the copy happened. 1 For those unfamiliar with the terminology, that doesn’t mean it glows in the dark, it means it “happens all at once”. It’s from Greek and means “indivisible” or “uncuttable”. 2 2 Atoms were called atoms when people thought that they were the smallest particle that stuff was made of. We now know that atoms are made of even smaller things, but the name has stuck. |
Paolo Fabio Zaino (28) 1882 posts |
XD, if you try to google for this, you’ll end-up in a different dimention where a bunch of folks, in recent years tried to talk of cooperative schedulers as an abstract entity, to be used to assign a task to a queue of tasks either on a GPU or to a CPU. Their assumption however (and often undocumented) is that a task is a “portion” of the entire work… now I let the native English speaking folks work out the relationship between the term portion and slice ;)
Minor correction here, slicing (as a term) has become popular on OS architecture books that were trying to describe preempty schedulers1 and their benefits, but the actual use of the term comes from the previous studies that were meant to improve cooperative scheduling and batch scheduling (you seems to be a reader of Tanenbaum’s books, so I am sure you have read all the chapters about batch scheduling, multi-users scheduling etc. in his books) Those studies broguh us to preemptive as the easiest and better form of scheduling, which takes a lot of responsabilities away from the application developer and put them on the kernel developer which is assumed to be more experienced (this because we wanted to have more people capable of writing software and so reduce the level of skills required in application development, later on we messed this up by inventing complex application framework, but this is another story!).
Sorry, this is another confusion, probably Tanenbaum is responsible for this. Clarification: a thread it’s an execution path (some eronously referes to threads as a portion, but if such term should be used and declared valid, then a thread should be considered a “vertical portion”, not an “horizontal portion” of a piece of logic as a slice is usually used to refer to). I am sorry if I sound pedentic here, but we need a clear term to identify a segment of logic that YELDs control at some point (not governed by the logic it has to execute, but by the time its taking to execute it), and threads are not such a clear term, because they are used to assign an execution path to a worker1, they don’t necessarly express the concept of breaking-up such path in chunks (or slices) that yeld every X instructions or every X milliseconds. I know this can generate some reaction from you, but please don’t take it personal. We really need a specific term to identify a segment of code where it’s execution time is well known even outside of his task or thread execution path. Am I making sense? The issue is that in a preemptive scheduler such a unit of work is managed by the kernel, while on RISC OS it’s managed by the Application developer (or the Module developer etc.).
Indeed. Actually I stand correct there, because in our case, at least for now, it’s only the window manager (as the cooperative scheduler lives in the window manager and not in the actual OS kernel, although if in RISC OS, “kernel” is a very mild term) 1 for the non trained: “scheduler” means a code that assign tasks to a worker, a worker in our case is usually identified as CPU, I have forgot to mention this on my previous post. |
Rick Murray (539) 13840 posts |
Hmmm… this doesn’t really have much to do with SQLite crashing…
“It depends”. If we’re talking cake, then a slice and a portion are probably the same. If we’re talking bread, then a slice is a single piece while it takes two to make a sandwich, which is the portion. English is full of terms that are all subtly similar but not quite. I think it’s better to think that portion >= slice; rather like long >= int. ;)
Certainly. The programmer can concentrate on getting the task doing what it does, and not have to jump through a bunch of hoops to satisfy the OS. It’s always good practice to yield as soon as you’re done, rather than finger twiddling until the time slice expires, but if you don’t the sky won’t fall, your first-born kittens won’t develop rabies, and your rivers won’t be full of effluent (unless you’re English). On the other hand, failing to yield at appropriate and regular times on co-operative is a problem. There’s more work that needs to be done to make an application play nice with others.
…agile, constant integration, waterfall, and other dumb metaphors that basically amount to “twiddle something, push it to the code, build it, treat your customers as unofficial testers”.
Hey, don’t bash the bloke. I read one book about twenty five years ago as “light reading material” after a hard day looking after dying old people. So, while I have an idea about processes, tasks, and threads, it’s not based upon what that sole author may or may not have said.
WTF are you on about vertical and horizontal? If we discard the complications of multiple execution units, multiple cores, and all the modern trickery… …a processor processes instructions one at a time. From the 6502 (and earlier) up to the latest ARM64 gizmo, the programmer or compiler lays out some instructions, and the processor crunches its way through them. A “process” is the program that is currently executing. In terms of RISC OS, this is known as a “task” because, really, the OS only understands a single process. Or, in simpler terms: Process means any program in execution. Thread means any segment of a process. I did a quick Google search for clarification, and that previous line was copied directly from GeeksForGeeks, but other sites say pretty much the same. I think you may be getting confused with vertical and horizontal scaling, but this is sort of like distributed systems and a kind of a server-class topic – namely scaling vertically means adding more to an existing machine (memory, processor cores, disc space) to handle increased loads, while scaling horizontally means adding additional machines to handle increased loads.
Do we? Outside of an RTOS which attempts to make guarantees about what will be available when, most operating systems use a “best effort” scheduler. Sure, there are many tweaks and fiddles, even RISC OS manages to have a high priority pollword, but really there’s no actual way to know when your task will be run again. On more competent systems, it is possible to set an execution priority level. My DVD ripper under XP runs with a “High” priority, so it’ll get… I’m not sure if it’s a larger time slice or a more frequent one, but it will get more time than everything else. [note to those who like fiddling – do not attempt to use the “realtime” priority option; that means that Windows will pretty much run that application to the exclusion of everything else, including itself… ask me how I know ;) ]
Schedulers are time based. Assuming no active locks, when a timer interrupt fires, the scheduler will put the current process/thread/whatever to sleep and look for a new one to shuffle into its place. It would be doomed to failure anyway, as while it was entirely possible to know exactly how long some code might take on the 6502 or the ARM2 (Z80, etc etc), these days with stuff like dual issue pipelines, speculative execution, out of order execution, and so forth, there’s literally no way of knowing to the cycle how long something will actually take to execute because it will depend upon the internal configuration of the processor (some of which can be updated with new microcode), so it would be a bit of a fool’s errand to even try. Of course, Linux being Linux, there are numerous scheduler algorithms available each attempting to provide “the best results” by widely varying definitions of “best”. Anyway, I don’t think we the application authors need to get too hung up on exactly when control will be handed back to us. Only that, at some point, it will. If we require precise and exact timing, we’d not be running an OS like typical Linux or Windows (or even RISC OS).
Well, it’s a good example of a monolithic “shove it all in here” kernel.
Which is why it’s better just to say “processor” or “core” (the latter being more common these days now that processors have many). Suffice to say, “at the end of the day” (argh! cliché!), it’s easier to get fluid responses from software running on a machine that has pre-emptive task switching as the kernel does most of the work. On co-operative, such as RISC OS, it’s the application writer that has to do the work. Actually, my first incarnation counted centiseconds and it got complex and messy in a hurry, so I tore all of that out and just polled at certain checkpoint positions. A “good enough” solution. While on a pre-emptive system, I just write the damn code and let the OS work out when to switch. |
Rick Murray (539) 13840 posts |
https://xkcd.com/386/ (or on mobile: https://m.xkcd.com/386/ ) :-) :-)
I don’t. Life’s too short to get upset about anything that isn’t:
and:
Mostly the last one. Which reminds me… time to go put the kettle on. ;) |
Paolo Fabio Zaino (28) 1882 posts |
No. The modern sources explains the concept of a thread ignoring the temporal segmentation that a process needs to have in order for “multi-tasking” to take place. In modern OS this is fine. A thread has never identified “a temporal slice/unit of work”, instead it identify a logical unit of work. Hence, it’s a concept that cannot be used to describe (again) a unit of work of a finite time that is required to describe the issue on a cooperative scheduler. As I said you can google as much as you want, but probably 99.99% of resources out there simply assume that the temporal slicing of a thread into code segments isn’t a problem that needs description and terminology, because (well) it has been solved by preemptive multitasking. :) On the other hand on a cooperative scheduler, we DO have this need, in order to determine the correct values for such a unit of work. Are we going to slice up per number of instructions executed? For instance “yeld every 10 opcodes of my process”, or yeld every 4 function calls, or yeld every 4 icons displayed on a window. Because that’s what we do now. Or, as you’ve mentioned, set a callback mechanism based on time, which could be used to set a global state and, within a for loop add a check if our internal global state says it’s time to yeld, and if so save the current internal state of the currently being processed event (that assume we have an internal queue) and return to the window manager. Or, as some software do, wrap app a commonly used function (for instance malloc) so that everytime it’s called we perform that feature after we yelded by calling a local Wimp_poll (which, however can have issues on RISC OS) Etc… If you desire to use the term threads for this, be my guest, but threads are an execution path, a process can be divided in multiple execution paths (and if GeeksForGeeks has a problem with this, they should go back to study how threads work, aka sharing global states between them etc.). With that out of the way, given that threads are generally rappresented by (well) threads, often as vertical lines, a temporal slice (to be well differentiated) should be an “horizontal” subdivision of a thread. If instead, one prefers to represent a thread as an horizontal line (never seens one, but hey it’s possible!), then the temporal slice should be represented as a vertical subdivision of that thread representation. In other words a temporal unit of work (aka a slice or whatever you want to call it) is not a Thread. A thread, terminate when it’s work is done. It’s has no temporal subdivision concept because that (again) is something that has been solved by preemptive scheduler and so, it’s no longer needed (technically never needed, ’cause threads came after preemptive multitasking). |
Paolo Fabio Zaino (28) 1882 posts |
The problem in topics was addressed long ago. The off-topic discussion started out of a comment that Steve made after I briefly mentioned why I no longer use SQLite3 with my code (which were meant to be: “I am not up to date with the latest changes ’cause I no longer use it for X reason” kinda thing). The crash with DDE is most likely caused by the way SQLite3 is being patched to compile with DDE. IIRC, SQLite uses GCC features like DLOpen and (possibly) others, so it’s possible that by trying to hide or work around those, new bugs were introduced. It’s something Chris Mahoney should look into (I think), but if it hasn’t been reported to him then… :) |
Paolo Fabio Zaino (28) 1882 posts |
Enjoy! :) |
Chris Mahoney (1684) 2165 posts |
I only just saw this. If you’re talking about “my” version of SQLite (here, and in PackMan) then I can say “it works on my machine” under DDE. I’ll have a play (maybe using a fresh install on another SD card) and see whether I can get it to fail, but I’m not sure when I’ll have the free time to actually do this. |
Paolo Fabio Zaino (28) 1882 posts |
@ Chris Mahoney
Just to be clear, yes both me and Michael are refering to your version of SQLite. In my case I have downloaded it through !PackMan and compiled with an example that works fine with the same version of SQLite provided by the gccsdk maintainer. However the GCCSDK provided one compiles only GCC 4 (or higher), it’s provided as a static ELF32 library. So, technically the only version that comes with AIF usable object fomat library is yours. When compiling a simple code with your and using latest DDE 31c, everything compiles fine, but as soon as we try to call functions in you SQLite3 library we get the following error: Internal error: branch through zero Postmortem requested 0 in unknown procedure c698 in anonymous function Arg4: 00000000 0 Arg3: 0x00000006 6 Arg2: 0x000b0750 722768 -> [00000000 0x000b0814 0x000aef5c 0x000af154] Arg1: 0x000081f0 33264 -> [0x65742e40 0x645f7473 0x00000062 0x000aed20] 9ff84 in anonymous function ... I did some debug for you and the crash happens exactly when we try to call sql3_open function. File is present in the local path and is SQLite 3 format. Opens and works fine with SQLite3 library for GCC. I ran my test on RISC OS 5.28 on RPCEmu (configured to emulate StrongARM + 256MB RAM, use hostFS, and file is on HostFS). HTH |
Paolo Fabio Zaino (28) 1882 posts |
Ok, digging more I have found that you seems not be using/have added an MkClean to your source build. This can cause troubles. DDE tend not to rebuild the dynamic dependencies if they are listed in your Makefile. So, cleaned up this and found the first issue with your build: AMU: command too long (must be shorter than 255 characters) cc 0zpz0 -c -depend !Depend -DSQLITE_OS_OTHER .... Which is a line composed by many macro definitions. This is an error that also ROOL got stuck with in their SHared Makefile for the debug term and it’s still in there with release 31c. I fixed the issues with the above and finally got it compiling from clean. However this define: #define SQLITE_HAS_ISNAN Seems to be causing a weird error during the final part of the compilation, so I need to look into that, but I am too tired now. In any case, disabling that #define makes it compile fully on all my test systems and from complete clean. I uploaded all the fixes here for you to have a look: https://github.com/RISC-OS-Community/SQLite3 I also mentioned in that repo that you still want people to download the source from your site, but it would be great if we could work together. HTH |
Chris Mahoney (1684) 2165 posts |
[Edit: Most of this was a red herring.] Ahh, yes, I see what’s happened. My tests (which you won’t have seen, as they’re not part of the distributed archives) need a bit more work. The official advice is that you include the c.sqlite3 “amalgamation” rather than the library, and therefore my test suite does the same thing. I did have a test around the compiled library at one point but it looks like I accidentally got rid of it! I’ll fix that up when I can :) Now I’m wondering how long this has been broken given that I’ve been doing fairly simple updates recently… but I’m too lazy to dig through my source control history and figure out when it broke.
True, but this “shouldn’t” be causing the problem, because the makefiles in source control don’t have the dynamic dependencies listed and I always compile from a clean checkout. I’ll double-check though.
Be warned: with that #define disabled, any databases using floating-point field types will not be compatible with non-RISC OS versions of SQLite, and vice versa. Sorry for the inconvenience! Edited to add: I think my (undocumented) thought process at the time, 18 months ago or whenever it was, was that developers would download the source archive, run Mk to build c.sqlite3, and then use the amalgamation directly in their projects. Meanwhile I put the precompiled library in the PackMan package so that developers could optionally use the library and get the advantage of automatic upgrades via PackMan. It’s clear that my test projects need to verify both methods, but they currently don’t. |
Chris Mahoney (1684) 2165 posts |
This is probably going to have to wait until the weekend. The precompiled library works fine in my existing test “harness” app, but not in a basic app. I’ve yet to figure out why, and I don’t fancy trying to figure it out after working all day :) |
Chris Mahoney (1684) 2165 posts |
I’m going to retract my previous statement: I was apparently half-asleep because I sat down to look at this properly today… and I can’t make it fail. Please download this test zip, run Mk, and see whether the compiled binary successfully creates “mydb” in the current directory. The libsqlite3 and h.sqlite3 files included in that zip are the same versions that are currently in PackMan, and it builds and runs correctly on my machine. Edit: Actually, in my test app from a couple of days ago I’d forgotten the call to |
Michael Grunditz (8594) 259 posts |
Not workng here. On Pi4 with local riscos build 5.29. EDIT.. I forgot to increase wimp slot! EDIT.. I forgot to increase wimp slot!It is working fine now :) |
Pages: 1 2