Contact: fumanchu@aminus.org

Log in as guest/dejavu to create tickets

root/tags/1.4.0/doc/storage.html

Revision 165 (checked in by fumanchu, 3 years ago)

Now distributing recur module with dejavu.

Line 
1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
2    "http://www.w3.org/TR/xhtml1/DTD/strict.dtd">
3 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
4
5 <head>
6     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
7     <title>Dejavu: Configuring Storage</title>
8     <link href='dejavu.css' rel='stylesheet' type='text/css' />
9 </head>
10
11 <body>
12
13 <h2>Deployers: Configuring Storage</h2>
14
15 <h3>Storage Managers</h3>
16
17 <p>Storage Managers insulate an application developer from the specifics of
18 databases, query languages, and cache mechanisms. As the <i>deployer</i> of
19 a Dejavu application, you get to be in control of these specifics. But
20 don't worry; in the vast majority of cases, you will set up a single
21 database with just two lines in a configuration file. Often, the
22 application developer will have already prepared default config files
23 which you can simply "plug and play". But if you <i>need</i> more control
24 over your data storage, you have it, without becoming a programmer.</p>
25
26 <p>When you deploy an app built with Dejavu, you must specify Storage
27 Managers to use for persisting application objects. This is usually
28 done through an ini-style configuration file. Here's a short example:
29 <pre>[Junct]
30 Class: dejavu.storage.storeado.StorageManagerADO_MSAccess
31 Connect: "PROVIDER=MICROSOFT.JET.OLEDB.4.0;DATA SOURCE=D:\data\junct.mdb;"
32 </pre>
33 The first line of our example ("[Junct]") names the Storage Manager;
34 each section in your conf file defines a different SM. You can use whatever
35 name you like here; in this example, we used the name of the application.
36 The second line tells Dejavu the <i>class</i> of SM we'd like to use.
37 For most applications, you'll decide which class to use based on the
38 database you want to use. Our example declares that we want to persist our
39 application data in an "MS Access" (i.e., Jet) database. The third line in
40 our example is a standard ADO Connect string. The _MSAccess class requires
41 this entry; other SM's may not.</p>
42
43 <h4>Common Configuration Entries</h4>
44 <p>There are a few configuration entries which (probably) apply to all
45 Storage Managers:</p>
46
47 <table>
48 <tr><th>Key</th><th>Example Value</th><th>Description</th></tr>
49 <tr>
50     <td>Class</td>
51     <td><tt>dejavu.storage.CachingProxy</tt></td>
52     <td>Class to use when instantiating this <tt>StorageManager</tt>.</td>
53 </tr>
54 <tr>
55     <td>Load Order</td>
56     <td><tt>5</tt></td>
57     <td>Optional. The order in which to load this SM. Lower numbers are
58         loaded first. SM's without a Load Order default to 0.</td>
59 </tr>
60 <tr>
61     <td>Shutdown Order</td>
62     <td><tt>10</tt></td>
63     <td>Optional. The order in which to shut down this SM. Lower numbers are
64         shut down first. SM's without a Shutdown Order default to 0.</td>
65 </tr>
66 <tr>
67     <td>Units</td>
68     <td><tt>[UnitCollection, UnitEngine, UnitEngineRule, FieldDashboardSumSet]</tt></td>
69     <td>Optional. Declares which Unit classes to manage with this SM
70         (see below).</td>
71 </tr>
72 </table>
73
74 <p>The "Units" entry is what you will use to separate application objects
75 into separate stores (if you need to). The objects in an application which
76 need to be stored are called "Units", and each Unit is of a certain Unit
77 class. If you specify a "Units" entry, then only Units of those classes
78 will be managed by that Storage Manager. If you do <i>not</i> specify such
79 an entry, then <b>all</b> Units will be handled by that Storage Manager.
80 This means that only <i>one</i> SM should be missing this entry.</p>
81
82
83 <h4>Database Storage Managers</h4>
84
85 <h5>Microsoft SQL Server / Microsoft Access (Jet)</h5>
86 <p>This module was developed against ADO 2.7. Configuration entries:</p>
87 <ul>
88     <li><b>Class:</b> <tt>dejavu.storage.storeado.StorageManagerADO_SQLServer</tt>
89         or <tt>dejavu.storage.storeado.StorageManagerADO_MSAccess</tt></li>
90     <li><b>Connect:</b> A valid ADO connect string. There are plenty of
91         online references for how to form these; for example, at
92         <a href='http://support.microsoft.com/?kbid=193332'>Microsoft</a>.</li>
93 </ul>
94
95 <h5>PostgreSQL (pyPgSQL)</h5>
96 <p>This class was developed against
97     PostgreSQL 8.0.0 rc-1 on Win2k,
98     and also tested on
99     PostgreSQL 7.6.6-6 on Debian "sarge".
100     Configuration entries:</p>
101 <ul>
102     <li><b>Class:</b> <tt>dejavu.storage.storepypgsql.StorageManagerPgSQL</tt></li>
103     <li><b>Connect:</b> A connect string of the form "k=v k=v". For example,
104         <tt>"host=localhost dbname=myapp user=postgres password=hilar1ous"</tt>.
105         See the <a href='http://www.postgresql.org/docs/current/static/libpq.html'>libpq</a>
106         docs for complete information.</li>
107 </ul>
108
109 <h5>MySQL (MySQLdb)</h5>
110 <p>This class was developed against
111     mysql  Ver 14.7 Distrib 4.1.8, for Win95/Win98 (i32),
112     and also tested on
113     mysql  Ver 12.22 Distrib 4.0.23, for pc-linux-gnu (i386).
114     Configuration entries:</p>
115 <ul>
116     <li><b>Class:</b> <tt>dejavu.storage.storemysql.StorageManagerMySQL</tt></li>
117     <li>Connection arguments: any of "host", "user", "passwd", "db", "port",
118         "unix_socket", "client_flag".<br />See the
119         <a href='http://dev.mysql.com/doc/mysql/en/mysql_real_connect.html'>docs</a>
120         for complete info.</li>
121 </ul>
122
123 <h5>SQLite (pysqlite)</h5>
124 <p>This class was developed against
125     sqlite 3.0.8 (pysqlite-1.1.6.win32-py2.3),
126     and also tested on
127     sqlite 3.3.3 (pysqlite-1.1.7.win32-py2.4),
128     sqlite 2.8.15-3 on Debian "sarge".
129     Configuration entries:</p>
130 <ul>
131     <li><b>Class:</b> <tt>dejavu.storage.storemysql.StorageManagerMySQL</tt></li>
132     <li><b>Database:</b> Filename of the database. May be a relative path.
133         If the DB does not already exist, it will be created.</li>
134     <li><b>Mode:</b> Optional. DB file mode. Defaults to 0755.</li>
135 </ul>
136
137 <h5>ODBC</h5>
138 <p>This class doesn't support create_database or drop_database. Actually,
139 it doesn't support much of anything; it's quite broken at present.
140 Configuration entries:</p>
141 <ul>
142     <li><b>Class:</b> <tt>dejavu.storage.storeodbc.StorageManagerODBC</tt></li>
143     <li><b>Connect:</b> A valid ODBC connect string.</li>
144 </ul>
145
146 <h5>Shelve</h5>
147 <p>Persists Units to shelve-type db files. Extremely simple implementation;
148 everything is pickled. Querying will be slow--every Unit is sucked in
149 one-by-one and tested in pure Python using <tt>Expression(unit)</tt>.
150 But for many applications, you don't need heavyweight query tools;
151 for example, an online forum may only need topic content looked up by ID.
152 Or small system tables that only get read at startup might benefit.
153 Configuration entries:</p>
154 <ul>
155     <li><b>Class:</b> <tt>dejavu.storage.storeshelve.StorageManagerShelve</tt></li>
156     <li><b>Path:</b> The file path (directory) in which to place db files.
157         Each Unit subclass will get its own file, of the same name as the
158         subclass.</li>
159 </ul>
160
161 <h5>Common Database Configuration Entries</h5>
162 <p>In addition to the above, Storage Managers for databases (probably)
163 accept these additional options:</p>
164
165 <table>
166 <tr><th>Key</th><th>Example Value</th><th>Description</th></tr>
167 <tr>
168     <td>Pool Size</td>
169     <td><tt>10</tt></td>
170     <td>Optional. Defaults to 10. If nonzero, connections will be pooled
171         (up to a total equal to <i>Pool Size</i>). If zero, no pool
172         will be used; each statement (!) will use a new connection.</td>
173 </tr>
174 <tr>
175     <td>Prefix</td>
176     <td><tt>myapp_</tt></td>
177     <td>Optional. If specified, all tables in the database will have names
178     starting with this prefix. If not provided, it defaults to "djv". This
179     helps if you need to mix Dejavu tables with tables from another
180     application. Set to blank if you want no prefix.</td>
181 </tr>
182 <tr>
183     <td>Create If Missing</td>
184     <td><tt>True</tt></td>
185     <td>Optional. Defaults to True (set to blank to turn off). If not blank,
186         create the database as needed. Because of the vagaries of various
187         databases, the ODBC Storage Manager doesn't support this.</td>
188 </tr>
189 <tr>
190     <td>Type Adapter</td>
191     <td><tt>myapp.storage.FieldTypeAdapterForMyDB</tt></td>
192     <td>Optional. The "Type Adapter" is used to map Python types to database
193         column types for use in <tt>CREATE TABLE</tt> statements; for
194         example, the Python <tt>float</tt> type might be mapped to a
195         <tt>REAL</tt> column type. If you don't like the default column
196         types which your Storage Manager provides, you can write your own
197         adapter and declare its use here. The value should be the full
198         dotted package name of the class you wish to use.</td>
199 </tr>
200 <tr>
201     <td>To Adapter</td>
202     <td><tt>myapp.storage.AdapterToMyDBSQL</tt></td>
203     <td>Optional. The "To Adapter" is used to map Python values to database
204         values for use in SQL statements; for example, the Python <tt>str</tt>
205         type usually needs to be wrapped in quote marks. If you don't like
206         the SQL which your Storage Manager generates, you can write your
207         own adapter and declare its use here. The value should be the full
208         dotted package name of the class you wish to use.</td>
209 </tr>
210 <tr>
211     <td>From Adapter</td>
212     <td><tt>myapp.storage.AdapterFromMyDB</tt></td>
213     <td>Optional. The "From Adapter" is used to map incoming database values
214         (i.e., the results of a <tt>SELECT</tt> query) to Python values; for
215         example, your database may return a date value as a string, which
216         must then be converted to the Python <tt>datetime.date</tt> type.
217         If you don't like the default coercions which your Storage Manager
218         provides, you can write your own adapter and declare its use here.
219         The value should be the full dotted package name of the class you
220         wish to use.</td>
221 </tr>
222 <tr>
223     <td>Expanded Columns</td>
224     <td><tt>Animal.PreviousZoos:int, Exhibit.Animals:int</tt></td>
225     <td>Optional. A comma-separated list of UnitClass.Property:subtype
226         strings. Each such property should be of .type list or tuple.
227         Usually, lists are pickled for storage in a normal database
228         field. Properties listed in <tt>Expanded Columns</tt> will
229         be stored each in their own table. The "subtype" portion tells
230         the Storage Manager the type of each value in the list. This is
231         mostly here to support a legacy database which already normalizes
232         the values in this way; for new projects, we recommend using the
233         default pickle method, which is much faster and more manageable.</td>
234 </tr>
235 </table>
236
237
238 <h4>Middleware</h4>
239
240 <p>Some Storage Managers act as "middleware", and can be chained together
241 to provide layered functionality. Consider, for example, the
242 <tt>CachingProxy</tt> class; it usually has another Storage Manager
243 "behind it", which it proxies. It can be used to cache objects between
244 client connections independently from the underlying, database-specific
245 Storage Manager. The beauty of this design is that the decision to
246 use a CachingProxy is completely up to the deployer, <i>not</i> the
247 application developer. The deployer can test response times, separate
248 stores, and address other integration concerns on their own systems.</p>
249
250 <h5>Caching Proxy</h5>
251 <p>Use this class to persist Units in memory between client connections.
252 It usually proxies another Storage Manager. Configuration entries:</p>
253 <ul>
254     <li><b>Class:</b> <tt>dejavu.storage.CachingProxy</tt></li>
255     <li><b>Next Store:</b> Optional. The name of the next Storage Manager
256         in the chain. If you do not specify a Next Store, Units will
257         only persist for the lifetime of the arena.</li>
258     <li><b>Lifetime:</b> Optional. The recurrence string which declares
259         how often to sweep Units out of the in-memory cache. The string you
260         supply should be one of the following types:
261         <ul>
262             <li><b>By units (intervals):</b> "3 hours" will run every 3
263                 hours. "7 days" or "1 week" will run once each week.</li>
264             <li><b>Daily:</b> "14:00 each day" will run at 2:00 P.M.
265                 every day.</li>
266             <li><b>Weekly:</b> "Mon", "Monday", or "Mondays" will run once
267                 each Monday.</li>
268             <li><b>Monthly:</b> "20 each month" will run on the 20th of
269                 each month. "0 every month" will run on the <i>last</i>
270                 day of each month.</li>
271         </ul>
272     </li>
273 </ul>
274
275
276 <h5>Burned Proxy</h5>
277 <p>Use this class to persist Units in memory between client connections.
278 It needs another Storage Manager to proxy. Unlike the Caching Proxy above,
279 this Storage Manager recalls all Units at once upon the first request,
280 and won't recall them again from storage. They are "burned" into memory
281 for the lifetime of the application. Configuration entries:</p>
282 <ul>
283     <li><b>Class:</b> <tt>dejavu.storage.BurnedProxy</tt></li>
284     <li><b>Next Store:</b> Required. The name of the next Storage Manager
285         in the chain.</li>
286     <li><b>Lifetime:</b> Optional. The recurrence string which declares
287         how often to sweep Units out of the in-memory cache. See the
288         Caching Proxy, above, for recurrence string formats. In general,
289         you should not set this value for BurnedProxy stores.</li>
290 </ul>
291
292 </body>
293 </html>
Note: See TracBrowser for help on using the browser.