Changesets can be listed by changeset number.
The Git repository is here.
- Revision:
- 15
- Log:
Attempt to update Typo to a Typo SVN HEAD release from around the
time the prototype installation was set up on the RISC OS Open Limited
web site. Timestamps place this at 04-Jul so a revision from 05-Jul or
earlier was pulled and copied over the 2.6.0 tarball stable code.
- Author:
- adh
- Date:
- Sat Jul 22 23:27:35 +0100 2006
- Size:
- 1329 Bytes
1 | Lexical analysis is performed by obtaining a tokenizer of the appropriate class and calling @tokenize@ on it, passing the text to be tokenized. Each token is yielded to the associated block as it is discovered. |
2 | |
3 | {{{lang=ruby,number=true,caption=Tokenizing a Ruby script |
4 | require 'syntax' |
5 | |
6 | tokenizer = Syntax.load "ruby" |
7 | tokenizer.tokenize( File.read( "program.rb" ) ) do |token| |
8 | puts token |
9 | puts " group: #{token.group}" |
10 | puts " instruction: #{token.instruction}" |
11 | end |
12 | }}} |
13 | |
14 | If you need finer control over the process, you can use the lower-level API: |
15 | |
16 | {{{lang=ruby,number=true,caption=Tokenizing a Ruby script via step |
17 | require 'syntax' |
18 | |
19 | tokenizer = Syntax.load "ruby" |
20 | tokenizer.start( File.read( "program.rb" ) ) do |token| |
21 | puts token |
22 | puts " group: #{token.group}" |
23 | puts " instruction: #{token.instruction}" |
24 | end |
25 | |
26 | tokenizer.step |
27 | tokenizer.step |
28 | ... |
29 | tokenizer.finish |
30 | }}} |
31 | |
32 | In this case, each time @#step@ is invoked, it results in tokens being consumed and yielded to the block. However, a single step may result in multiple tokens being detected and yielded--there is no way to guarantee a single token at a time, unless the corresponding syntax module was written to work that way. For efficiency, the existing modules will yield multiple tokens when processing (for instance) strings, regular expressions, and heredocs. |