Changeset 61
- Timestamp:
- 01/24/05 05:03:16
- Files:
-
- trunk/doc/index.html (modified) (2 diffs)
- trunk/doc/storage.html (modified) (9 diffs)
- trunk/engines.py (modified) (2 diffs)
- trunk/storage/db.py (modified) (1 diff)
- trunk/storage/storeado.py (modified) (6 diffs)
- trunk/storage/test_storage.py (added)
Legend:
- Unmodified
- Added
- Removed
- Modified
- Copied
- Moved
trunk/doc/index.html
r60 r61 89 89 <li>Storage Managers 90 90 <ul> 91 <li>Common Configuration Entries</li> 91 92 <li>Database Storage Managers 92 93 <ul> … … 97 98 <li>ODBC</li> 98 99 <li>Shelve</li> 100 <li>Common Database Configuration Entries</li> 99 101 </ul> 100 102 </li> trunk/doc/storage.html
r60 r61 29 29 <pre>[Junct] 30 30 Class: dejavu.storage.storeado.StorageManagerADO_MSAccess 31 Connect: " DSN=Junct;Driver=Microsoft Access Driver (*.mdb);Dbq=D:\data\junct.mdb;"31 Connect: "PROVIDER=MICROSOFT.JET.OLEDB.4.0;DATA SOURCE=D:\data\junct.mdb;" 32 32 </pre> 33 33 The first line of our example ("[Junct]") names the Storage Manager; … … 37 37 For most applications, you'll decide which class to use based on the 38 38 database you want to use. Our example declares that we want to persist our 39 application data in an "MS Access" (i.e., Jet) database.</p> 40 41 <p>The third line in our example is a standard ODBC-style Connect string. 42 The _MSAccess class requires this entry; other SM's may not. There are a few 43 configuration entries which (probably) apply to all Storage Managers:</p> 39 application data in an "MS Access" (i.e., Jet) database. The third line in 40 our example is a standard ADO Connect string. The _MSAccess class requires 41 this entry; other SM's may not.</p> 42 43 <h4>Common Configuration Entries</h4> 44 <p>There are a few configuration entries which (probably) apply to all 45 Storage Managers:</p> 44 46 45 47 <table> … … 67 69 <td>Optional. Declares which Unit classes to manage with this SM 68 70 (see below).</td> 69 </tr>70 <tr>71 <td>Pool Size</td>72 <td><tt>10</tt></td>73 <td>Optional. Defaults to 10. If nonzero, connections will be pooled74 (up to a total equal to <i>Pool Size</i>). If zero, no pool75 will be used; each statement (!) will use a new connection.</td>76 </tr>77 <tr>78 <td>Prefix</td>79 <td><tt>myapp_</tt></td>80 <td>Optional. If specified, all tables in the database will have names81 starting with this prefix. If not provided, it defaults to "djv". This82 helps if you need to mix Dejavu tables with tables from another83 application. Set to blank if you want no prefix.</td>84 </tr>85 <tr>86 <td>Create If Missing</td>87 <td><tt>True</tt></td>88 <td>Optional. Defaults to True (set to blank to turn off). If not blank,89 create the database as needed. Because of the vagaries of various90 databases, the ODBC Storage Manager doesn't support this.</td>91 </tr>92 <tr>93 <td>Expanded Columns</td>94 <td><tt>Animal.PreviousZoos:int, Exhibit.Animals:int</tt></td>95 <td>Optional. A comma-separated list of UnitClass.Property:subtype96 strings. Each such property should be of .type list or tuple.97 Usually, lists are pickled for storage in a normal database98 field. Properties listed in <tt>Expanded Columns</tt> will99 be stored each in their own table. The "subtype" portion tells100 the Storage Manager the type of each value in the list.</td>101 71 </tr> 102 72 </table> … … 121 91 online references for how to form these; for example, at 122 92 <a href='http://support.microsoft.com/?kbid=193332'>Microsoft</a>.</li> 123 <li><b>CursorType:</b> Optional. Passed to Recordset.Open(). See124 <a href='http://msdn.microsoft.com/library/en-us/ado270/htm/mdmthrstopen.asp'>Microsoft</a>125 again.</li>126 <li><b>LockType:</b> Optional. Passed to Recordset.Open(). See127 <a href='http://msdn.microsoft.com/library/en-us/ado270/htm/mdmthrstopen.asp'>Microsoft</a>128 again.</li>129 93 </ul> 130 94 … … 144 108 145 109 <h5>MySQL (MySQLdb)</h5> 146 <p>This class was developed against :110 <p>This class was developed against 147 111 mysql Ver 14.7 Distrib 4.1.8, for Win95/Win98 (i32), 148 112 and also tested on … … 158 122 159 123 <h5>SQLite (pysqlite)</h5> 160 <p>This class was developed against :124 <p>This class was developed against 161 125 sqlite 3.0.8 (pysqlite-1.1.6.win32-py2.3), 162 126 and also tested on … … 193 157 </ul> 194 158 <p>The shelve module does not yet support multirecall().</p> 159 160 <h5>Common Database Configuration Entries</h5> 161 <p>In addition to the above, Storage Managers for databases (probably) 162 accept these additional options:</p> 163 164 <table> 165 <tr><th>Key</th><th>Example Value</th><th>Description</th></tr> 166 <tr> 167 <td>Pool Size</td> 168 <td><tt>10</tt></td> 169 <td>Optional. Defaults to 10. If nonzero, connections will be pooled 170 (up to a total equal to <i>Pool Size</i>). If zero, no pool 171 will be used; each statement (!) will use a new connection.</td> 172 </tr> 173 <tr> 174 <td>Prefix</td> 175 <td><tt>myapp_</tt></td> 176 <td>Optional. If specified, all tables in the database will have names 177 starting with this prefix. If not provided, it defaults to "djv". This 178 helps if you need to mix Dejavu tables with tables from another 179 application. Set to blank if you want no prefix.</td> 180 </tr> 181 <tr> 182 <td>Create If Missing</td> 183 <td><tt>True</tt></td> 184 <td>Optional. Defaults to True (set to blank to turn off). If not blank, 185 create the database as needed. Because of the vagaries of various 186 databases, the ODBC Storage Manager doesn't support this.</td> 187 </tr> 188 <tr> 189 <td>Type Adapter</td> 190 <td><tt>myapp.storage.FieldTypeAdapterForMyDB</tt></td> 191 <td>Optional. The "Type Adapter" is used to map Python types to database 192 column types for use in <tt>CREATE TABLE</tt> statements; for 193 example, the Python <tt>float</tt> type might be mapped to a 194 <tt>REAL</tt> column type. If you don't like the default column 195 types which your Storage Manager provides, you can write your own 196 adapter and declare its use here. The value should be the full 197 dotted package name of the class you wish to use.</td> 198 </tr> 199 <tr> 200 <td>To Adapter</td> 201 <td><tt>myapp.storage.AdapterToMyDBSQL</tt></td> 202 <td>Optional. The "To Adapter" is used to map Python values to database 203 values for use in SQL statements; for example, the Python <tt>str</tt> 204 type usually needs to be wrapped in quote marks. If you don't like 205 the SQL which your Storage Manager generates, you can write your 206 own adapter and declare its use here. The value should be the full 207 dotted package name of the class you wish to use.</td> 208 </tr> 209 <tr> 210 <td>From Adapter</td> 211 <td><tt>myapp.storage.AdapterFromMyDB</tt></td> 212 <td>Optional. The "From Adapter" is used to map incoming database values 213 (i.e., the results of a <tt>SELECT</tt> query) to Python values; for 214 example, your database may return a date value as a string, which 215 must then be converted to the Python <tt>datetime.date</tt> type. 216 If you don't like the default coercions which your Storage Manager 217 provides, you can write your own adapter and declare its use here. 218 The value should be the full dotted package name of the class you 219 wish to use.</td> 220 </tr> 221 <tr> 222 <td>Expanded Columns</td> 223 <td><tt>Animal.PreviousZoos:int, Exhibit.Animals:int</tt></td> 224 <td>Optional. A comma-separated list of UnitClass.Property:subtype 225 strings. Each such property should be of .type list or tuple. 226 Usually, lists are pickled for storage in a normal database 227 field. Properties listed in <tt>Expanded Columns</tt> will 228 be stored each in their own table. The "subtype" portion tells 229 the Storage Manager the type of each value in the list. This is 230 mostly here to support a legacy database which already normalizes 231 the values in this way; for new projects, we recommend using the 232 default pickle method, which is much faster and more manageable.</td> 233 </tr> 234 </table> 235 195 236 196 237 <h4>Middleware</h4> … … 216 257 only persist for the lifetime of the arena.</li> 217 258 <li><b>Lifetime:</b> Optional. The recurrence string which declares 218 how often to sweep Units out of the in-memory cache.</li> 219 </ul> 259 how often to sweep Units out of the in-memory cache. If you supply 260 this value, you need to grab the <tt>recur</tt> module from 261 <tt>svn://casadeamor.com/misc/trunk</tt>. The string you supply 262 should be one of the following types: 263 <ul> 264 <li><b>By units (intervals):</b> "3 hours" will run every 3 265 hours. "7 days" or "1 week" will run once each week.</li> 266 <li><b>Daily:</b> "14:00 each day" will run at 2:00 P.M. 267 every day.</li> 268 <li><b>Weekly:</b> "Mon", "Monday", or "Mondays" will run once 269 each Monday.</li> 270 <li><b>Monthly:</b> "20 each month" will run on the 20th of 271 each month. "0 every month" will run on the <i>last</i> 272 day of each month.</li> 273 </ul> 274 </li> 275 </ul> 276 220 277 221 278 <h5>Burned Proxy</h5> … … 231 288 in the chain.</li> 232 289 <li><b>Lifetime:</b> Optional. The recurrence string which declares 233 how often to sweep Units out of the in-memory cache. In general, 290 how often to sweep Units out of the in-memory cache. See the 291 Caching Proxy, above, for recurrence string formats. In general, 234 292 you should not set this value for BurnedProxy stores.</li> 235 293 </ul> 236 237 <h4>Recurrence Values</h4>238 <p>In the above Storage Managers, you might be asked to supply a "recurrence239 string", which specifies a schedule for a given action. The string you240 supply should be one of the following types:</p>241 <ul>242 <li><b>By units (intervals):</b> "3 hours" will run every 3 hours.243 "7 days" or "1 week" will run once each week.</li>244 <li><b>Daily:</b> "14:00 each day" will run at 2:00 P.M. every day.</li>245 <li><b>Weekly:</b> "Mon", "Monday", or "Mondays" will run once each246 Monday.</li>247 <li><b>Monthly:</b> "20 each month" will run on the 20th of each month.248 "0 every month" will run on the <i>last</i> day of each month.</li>249 </ul>250 251 294 252 295 </body> trunk/engines.py
r60 r61 166 166 """kw: Operation, SetID, Operand=(Type | logic.Expression | otherSet) 167 167 168 TRANSFORM: 169 If the Operation is 'TRANSFORM', the Operand shall be the name 170 of a Unit type. The snapshot will consist of IDs of all units 171 of that Type which are associated with the current snapshot. 172 168 173 FILTER: 169 174 If the Operation is 'FILTER', the Operand shall be a 170 175 logic.Expression, and the snapshot will consist of the IDs of 171 176 Units which match the Expression. 172 173 Everything else:174 transforms: the snapshot will consist of IDs of all units175 which are associated with the current snapshot.176 union, difference, and intersection: these all take a setID.177 177 178 178 So, a typical Engine might have a set of rules which look like: … … 533 533 534 534 def visit_TRANSFORM(self, setID, operand): 535 """ operand=far class name. Multiple hops are supported."""535 """visit_TRANSFORM(setID, operand=farClass name). Multiple hops OK.""" 536 536 A = self.sets[setID] 537 537 start = self.arena.class_by_name(A.Type) trunk/storage/db.py
r60 r61 681 681 def __init__(self, name, arena, allOptions={}): 682 682 storage.StorageManager.__init__(self, name, arena, allOptions) 683 684 # Adapter Overrides 685 def get_adapter_option(name): 686 adapter_class = allOptions.get(name) 687 if isinstance(adapter_class, basestring): 688 import xray 689 adapter_class = xray.classes(adapter_class) 690 return adapter_class 691 692 adapter = get_adapter_option(u'Type Adapter') 693 if adapter: self.typeAdapter = adapter 694 adapter = get_adapter_option(u'To Adapter') 695 if adapter: self.toAdapter = adapter 696 adapter = get_adapter_option(u'From Adapter') 697 if adapter: self.fromAdapter = adapter 683 698 684 699 self.CreateIfMissing = bool(allOptions.get(u'Create If Missing', 'True')) trunk/storage/storeado.py
r60 r61 34 34 from dejavu import storage, logic 35 35 from dejavu.storage import db 36 from recur import sane_time37 36 38 37 adOpenForwardOnly = 0 … … 53 52 54 53 def time_from_com(com_date): 55 """Return a valid (day, datetime.time)from a COM date or time object."""56 hour, min s= divmod(86400 * (float(com_date) % 1), 3600)57 min s, sec = divmod(mins, 60)54 """Return a valid datetime.time from a COM date or time object.""" 55 hour, minute = divmod(86400 * (float(com_date) % 1), 3600) 56 minute, second = divmod(minute, 60) 58 57 # Must do both int() and round() or we'll be up to 1 second off. 59 58 hour = int(round(hour)) 60 mins = int(round(mins)) 61 sec = int(round(sec)) 62 return sane_time(0, hour, mins, sec) 59 minute = int(round(minute)) 60 second = int(round(second)) 61 62 while second > 59: 63 second -= 60 64 minute += 1 65 while second < 0: 66 second += 60 67 minute -= 1 68 while minute > 59: 69 minute -= 60 70 hour += 1 71 while minute < 0: 72 minute += 60 73 hour -= 1 74 while hour > 23: 75 hour -= 24 76 day += 1 77 while hour < 0: 78 hour += 24 79 80 return datetime.time(hour, minute, second) 63 81 64 82 … … 77 95 # For some reason, we need both float and int. 78 96 aDate = datetime.date.fromordinal(int(float(value)) + zeroHour) 79 day, aTime = time_from_com(value) 80 return datetime.datetime.combine(aDate, aTime) 97 return datetime.datetime.combine(aDate, time_from_com(value)) 81 98 82 99 def coerce_datetime_date(self, value, coltype): … … 90 107 def coerce_datetime_time(self, value, coltype): 91 108 # See coerce_datetime 92 day, aTime = time_from_com(value) 93 return aTime 109 return time_from_com(value) 94 110 95 111 def coerce_fixedpoint_FixedPoint(self, value, coltype): … … 444 460 atoms = self.connatoms() 445 461 self.dbname = atoms[u'INITIAL CATALOG'] 446 self.cursorType = int(allOptions.get(u'CursorType', adOpenDynamic))447 self.lockType = int(allOptions.get(u'LockType', adLockOptimistic))448 462 449 463 def create_database(self): … … 604 618 atoms.get(u'DATA SOURCE NAME') or 605 619 atoms.get(u'DBQ')) 606 self.cursorType = int(allOptions.get(u'CursorType', adOpenDynamic))607 self.lockType = int(allOptions.get(u'LockType', adLockOptimistic))608 620 # MS Access can't use a pool, because there doesn't seem 609 621 # to be a commit timeout.
