Changesets can be listed by changeset number.
The Git repository is here.
- Revision:
- 15
- Log:
Attempt to update Typo to a Typo SVN HEAD release from around the
time the prototype installation was set up on the RISC OS Open Limited
web site. Timestamps place this at 04-Jul so a revision from 05-Jul or
earlier was pulled and copied over the 2.6.0 tarball stable code.
- Author:
- adh
- Date:
- Sat Jul 22 23:27:35 +0100 2006
- Size:
- 11855 Bytes
1 | <html> |
2 | <head> |
3 | <title>Syntax Manual :: Chapter 4: Extending Syntax</title> |
4 | <link type="text/css" rel="stylesheet" href="stylesheets/manual.css" /> |
5 | </head> |
6 | |
7 | <body> |
8 | <div id="banner"> |
9 | <table border='0' cellpadding='0' cellspacing='0' width='100%'> |
10 | <tr><td valign='top' align='left'> |
11 | <div class="title"> |
12 | <span class="product">Syntax—</span><br /> |
13 | <span class="tagline">Lexical Analysis for Syntax Highlighting</span> |
14 | </div> |
15 | </td><td valign='middle' align='right'> |
16 | <div class="info"> |
17 | Syntax Version: <strong>1.0.0</strong><br /> |
18 | Manual Last Updated: <strong>2005-06-18 20:25 UTC</strong> |
19 | </div> |
20 | </td></tr> |
21 | </table> |
22 | </div> |
23 | |
24 | <table border='0' width='100%' cellpadding='0' cellspacing='0'> |
25 | <tr><td valign='top'> |
26 | |
27 | <div id="navigation"> |
28 | <h1>Syntax Manual</h1> |
29 | |
30 | <h2>Chapters</h2> |
31 | <ol type="I"> |
32 | |
33 | <li> |
34 | <a href="chapter-1.html"> |
35 | Introduction |
36 | </a> |
37 | |
38 | <ol type="1"> |
39 | |
40 | <li><a href="chapter-1.html#s1">What is Syntax?</a></li> |
41 | |
42 | <li><a href="chapter-1.html#s2">Quick Start</a></li> |
43 | |
44 | </ol> |
45 | </li> |
46 | |
47 | <li> |
48 | <a href="chapter-2.html"> |
49 | Lexical Analysis |
50 | </a> |
51 | |
52 | <ol type="1"> |
53 | |
54 | <li><a href="chapter-2.html#s1">Groups</a></li> |
55 | |
56 | <li><a href="chapter-2.html#s2">Instructions</a></li> |
57 | |
58 | <li><a href="chapter-2.html#s3">Analyzing</a></li> |
59 | |
60 | </ol> |
61 | </li> |
62 | |
63 | <li> |
64 | <a href="chapter-3.html"> |
65 | Syntax Highlighting |
66 | </a> |
67 | |
68 | <ol type="1"> |
69 | |
70 | <li><a href="chapter-3.html#s1">Converting Text</a></li> |
71 | |
72 | <li><a href="chapter-3.html#s2">Custom Highlighters</a></li> |
73 | |
74 | </ol> |
75 | </li> |
76 | |
77 | <li><strong> |
78 | <a href="chapter-4.html"> |
79 | Extending Syntax |
80 | </a> |
81 | </strong> <big>←</big> |
82 | <ol type="1"> |
83 | |
84 | <li><a href="chapter-4.html#s1">Introduction</a></li> |
85 | |
86 | <li><a href="chapter-4.html#s2">Interface</a></li> |
87 | |
88 | <li><a href="chapter-4.html#s3">Scanning <span class="caps">API</span></a></li> |
89 | |
90 | <li><a href="chapter-4.html#s4">Registering Your New Syntax</a></li> |
91 | |
92 | </ol> |
93 | </li> |
94 | |
95 | </ol> |
96 | |
97 | <h2>Other Documentation</h2> |
98 | |
99 | <ul> |
100 | <li><a href="http://net-ssh.rubyforge.org/api/index.html">Net::SSH API</a></li> |
101 | <li><a href="http://rubyforge.org/tracker/?atid=1842&group_id=274&func=browse">Net::SSH FAQ</a></li> |
102 | </ul> |
103 | |
104 | <h2>Tutorials</h2> |
105 | <ol> |
106 | |
107 | </ol> |
108 | |
109 | <p align="center"><strong>More To Come...</strong></p> |
110 | |
111 | <div class="license"> |
112 | <a href="http://creativecommons.org/licenses/by-sa/2.0/"><img alt="Creative Commons License" border="0" src="http://creativecommons.org/images/public/somerights" /></a><br /> |
113 | This manual is licensed under a <a href="http://creativecommons.org/licenses/by-sa/2.0/">Creative Commons License</a>. |
114 | </div> |
115 | </div> |
116 | |
117 | </td><td valign='top' width="100%"> |
118 | |
119 | <div id="content"> |
120 | |
121 | <div class="top"><div class="prevnext"> |
122 | |
123 | <a href="chapter-3.html">Previous (3. Syntax Highlighting)</a> | |
124 | |
125 | <a href="index.html">Up</a> |
126 | |
127 | </div></div> |
128 | |
129 | <h1>4. Extending Syntax</h1> |
130 | |
131 | |
132 | |
133 | <h2> |
134 | <a name="s1"></a> |
135 | 4.1. Introduction |
136 | </h2> |
137 | |
138 | |
139 | |
140 | <div class="section"> |
141 | <p>Because of Syntax’s modular design, it is pretty straightforward to create your own syntax modules. The hardest part is doing the actual tokenizing of your chosen syntax.</p> |
142 | |
143 | |
144 | <p>You can use the existing syntax modules to guide your own implementation if you wish, but note that each module will have to solve unique problems, because of the uniqueness of each different syntax.</p> |
145 | </div> |
146 | |
147 | |
148 | |
149 | <h2> |
150 | <a name="s2"></a> |
151 | 4.2. Interface |
152 | </h2> |
153 | |
154 | |
155 | |
156 | <div class="section"> |
157 | <p>Your new syntax implementation should extend <code>Syntax::Tokenizer</code>—this sets up a rich domain-specific language for scanning and tokenizing.</p> |
158 | |
159 | |
160 | <p>Then, all you need to implement is the <code>#step</code> method, which should take no parameters. Each invocation of <code>#step</code> should extract at least one token, but may extract as many as you need it to. (Fewer is generally better, though.)</p> |
161 | |
162 | |
163 | <p>Additionally, you may also implement <code>#setup</code>, to perform any initialization that should occur when tokenizing begins. Similarly, <code>#teardown</code> may be implemented to do any cleanup that is needed.</p> |
164 | </div> |
165 | |
166 | |
167 | |
168 | <h2> |
169 | <a name="s3"></a> |
170 | 4.3. Scanning <span class="caps">API</span> |
171 | </h2> |
172 | |
173 | |
174 | |
175 | <div class="section"> |
176 | <p>Within a tokenizer, you have access to a rich set of methods for scanning the text. These methods correspond to the methods of the StringScanner class (i.e., <code>scan</code>, <code>scan_until</code>, <code>bol?</code>, etc.).</p> |
177 | |
178 | |
179 | <p>Additionally, subgroups of recent regexps (used in <code>scan</code>, etc.) can be obtained via <code>subgroup</code>, which takes as a parameter the group you want to query.</p> |
180 | |
181 | |
182 | <p>Tokenizing proceeds as follows:</p> |
183 | |
184 | |
185 | <ol> |
186 | <li>Identify a token (using <code>#peek</code>, <code>#scan</code>, etc.).</li> |
187 | <li>Start a new token group (using <code>#start_group</code>, passing the symbol for the group and optionally any text you want to seed the group with).</li> |
188 | <li>Append text to the current group either with additional calls to <code>#start_group</code> using the same group, or with <code>#append</code> (which just takes the text to append to the current group)</li> |
189 | </ol> |
190 | |
191 | |
192 | <p>Instead of <code>#start_group</code>, you can also use <code>#start_region</code>, which begins a new region for the given group, and <code>#end_region</code>, which closes the region.</p> |
193 | |
194 | |
195 | <p>Here is an example of a very, very simple tokenizer, that simple extracts words and numbers from the text:</p> |
196 | |
197 | |
198 | <div class='figure'> |
199 | <span class='caption'>Simple tokenizer [ruby]</span> |
200 | <div class='body'><table border='0' cellpadding='0' cellspacing='0'><tr><td class='lineno'>1<br />2<br />3<br />4<br />5<br />6<br />7<br />8<br />9<br />10<br />11<br />12<br />13<br /></td><td width='100%'><link rel='stylesheet' type='text/css' href='stylesheets/ruby.css' /><div class='ruby'><pre><span class="ident">require</span> <span class="punct">'</span><span class="string">syntax</span><span class="punct">'</span> |
201 | |
202 | <span class="keyword">class </span><span class="class">SimpleTokenizer</span> <span class="punct"><</span> <span class="constant">Syntax</span><span class="punct">::</span><span class="constant">Tokenizer</span> |
203 | <span class="keyword">def </span><span class="method">step</span> |
204 | <span class="keyword">if</span> <span class="ident">digits</span> <span class="punct">=</span> <span class="ident">scan</span><span class="punct">(/</span><span class="regex"><span class="escape">\d</span>+</span><span class="punct">/)</span> |
205 | <span class="ident">start_group</span> <span class="symbol">:digits</span><span class="punct">,</span> <span class="ident">digits</span> |
206 | <span class="keyword">elsif</span> <span class="ident">words</span> <span class="punct">=</span> <span class="ident">scan</span><span class="punct">(/</span><span class="regex"><span class="escape">\w</span>+</span><span class="punct">/)</span> |
207 | <span class="ident">start_group</span> <span class="symbol">:words</span><span class="punct">,</span> <span class="ident">words</span> |
208 | <span class="keyword">else</span> |
209 | <span class="ident">start_group</span> <span class="symbol">:normal</span><span class="punct">,</span> <span class="ident">scan</span><span class="punct">(/</span><span class="regex">.</span><span class="punct">/)</span> |
210 | <span class="keyword">end</span> |
211 | <span class="keyword">end</span> |
212 | <span class="keyword">end</span></pre></div></td></tr></table></div></div> |
213 | </div> |
214 | |
215 | |
216 | |
217 | <h2> |
218 | <a name="s4"></a> |
219 | 4.4. Registering Your New Syntax |
220 | </h2> |
221 | |
222 | |
223 | |
224 | <div class="section"> |
225 | <p>Once you’ve written your new syntax module, you need to register it with the Syntax library so that it can be found and used by the framework. To do this, just add it to the <code>Syntax::SYNTAX</code> hash:</p> |
226 | |
227 | |
228 | <div class='figure'> |
229 | <span class='caption'>Registering a new syntax [ruby]</span> |
230 | <div class='body'><table border='0' cellpadding='0' cellspacing='0'><tr><td class='lineno'>1<br />2<br />3<br />4<br />5<br />6<br />7<br /></td><td width='100%'><link rel='stylesheet' type='text/css' href='stylesheets/ruby.css' /><div class='ruby'><pre><span class="ident">require</span> <span class="punct">'</span><span class="string">syntax</span><span class="punct">'</span> |
231 | |
232 | <span class="keyword">class </span><span class="class">SimpleTokenizer</span> <span class="punct"><</span> <span class="constant">Syntax</span><span class="punct">::</span><span class="constant">Tokenizer</span> |
233 | <span class="punct">...</span> |
234 | <span class="keyword">end</span> |
235 | |
236 | <span class="constant">Syntax</span><span class="punct">::</span><span class="constant">SYNTAX</span><span class="punct">['</span><span class="string">simple</span><span class="punct">']</span> <span class="punct">=</span> <span class="constant">SimpleTokenizer</span></pre></div></td></tr></table></div></div> |
237 | |
238 | |
239 | <p>That’s it! Once you’ve done that, you can now use your syntax just by requiring the file that defines it, and then using the standard Syntax framework methods:</p> |
240 | |
241 | |
242 | <div class='figure'> |
243 | <span class='caption'>Using your new syntax [ruby]</span> |
244 | <div class='body'><table border='0' cellpadding='0' cellspacing='0'><tr><td class='lineno'>1<br />2<br />3<br />4<br />5<br /></td><td width='100%'><link rel='stylesheet' type='text/css' href='stylesheets/ruby.css' /><div class='ruby'><pre><span class="ident">require</span> <span class="punct">'</span><span class="string">simple-tokenizer</span><span class="punct">'</span> |
245 | <span class="ident">require</span> <span class="punct">'</span><span class="string">syntax/convertor/html</span><span class="punct">'</span> |
246 | |
247 | <span class="ident">convertor</span> <span class="punct">=</span> <span class="constant">Syntax</span><span class="punct">::</span><span class="constant">Convertors</span><span class="punct">::</span><span class="constant">HTML</span><span class="punct">.</span><span class="ident">for_syntax</span> <span class="punct">"</span><span class="string">simple</span><span class="punct">"</span> |
248 | <span class="ident">puts</span> <span class="ident">convertor</span><span class="punct">.</span><span class="ident">convert</span><span class="punct">(</span> <span class="punct">"</span><span class="string">hello 15 worlds!</span><span class="punct">"</span> <span class="punct">)</span></pre></div></td></tr></table></div></div> |
249 | </div> |
250 | |
251 | |
252 | |
253 | <div class="bottom"><div class="prevnext"> |
254 | |
255 | <a href="chapter-3.html">Previous (3. Syntax Highlighting)</a> | |
256 | |
257 | <a href="index.html">Up</a> |
258 | |
259 | </div></div> |
260 | |
261 | |
262 | </div> |
263 | |
264 | </td></tr> |
265 | </table> |
266 | </body> |
267 | </html> |