Changesets can be listed by changeset number.
The Git repository is here.
- Revision:
- 15
- Log:
Attempt to update Typo to a Typo SVN HEAD release from around the
time the prototype installation was set up on the RISC OS Open Limited
web site. Timestamps place this at 04-Jul so a revision from 05-Jul or
earlier was pulled and copied over the 2.6.0 tarball stable code.
- Author:
- adh
- Date:
- Sat Jul 22 23:27:35 +0100 2006
- Size:
- 1329 Bytes
1 | Within a tokenizer, you have access to a rich set of methods for scanning the text. These methods correspond to the methods of the StringScanner class (i.e., @scan@, @scan_until@, @bol?@, etc.). |
2 | |
3 | Additionally, subgroups of recent regexps (used in @scan@, etc.) can be obtained via @subgroup@, which takes as a parameter the group you want to query. |
4 | |
5 | Tokenizing proceeds as follows: |
6 | |
7 | # Identify a token (using @#peek@, @#scan@, etc.). |
8 | # Start a new token group (using @#start_group@, passing the symbol for the group and optionally any text you want to seed the group with). |
9 | # Append text to the current group either with additional calls to @#start_group@ using the same group, or with @#append@ (which just takes the text to append to the current group) |
10 | |
11 | Instead of @#start_group@, you can also use @#start_region@, which begins a new region for the given group, and @#end_region@, which closes the region. |
12 | |
13 | Here is an example of a very, very simple tokenizer, that simple extracts words and numbers from the text: |
14 | |
15 | {{{lang=ruby,number=true,caption=Simple tokenizer |
16 | require 'syntax' |
17 | |
18 | class SimpleTokenizer < Syntax::Tokenizer |
19 | def step |
20 | if digits = scan(/\d+/) |
21 | start_group :digits, digits |
22 | elsif words = scan(/\w+/) |
23 | start_group :words, words |
24 | else |
25 | start_group :normal, scan(/./) |
26 | end |
27 | end |
28 | end |
29 | }}} |