Contact: fumanchu@aminus.org

Log in as guest/dejavu to create tickets

I think I've seen this ORM somewhere before...

Changeset 46

Show
Ignore:
Timestamp:
01/02/05 06:53:53
Author:
fumanchu
Message:

1. Changed AdapterFromDB to take unit arg only on consume.
2. Abstracted SM's into db.StorageManagerDB base class. All stores now have save_expanded functionality.
3. Fixed storeodbc. Finally.
4. Added decimal.Decimal to adapters.
5. New precision, scale hints. db.FieldTypeAdapter? has new numeric_max_precision attribute.
6. Broke chunks of modeling.html out into managing.html.
7. New tests in zoo_fixture.

Files:

Legend:

Unmodified
Added
Removed
Modified
Copied
Moved

  • trunk/doc/framework.html

    r45 r46  
    220220lifting of the tests is done in zoo_fixture.</p> 
    221221 
    222 <p>Finally, there is a <tt>storeodbc</tt> module in the <tt>storage</tt> 
    223 package. It didn't keep pace with development; it's about &frac34; finished. 
    224 Feel free to finish it for me. ;)</p> 
    225  
    226222<h4>Other Serialization Mechanisms</h4> 
    227223<h5>sockets</h5> 

  • trunk/doc/index.html

    r45 r46  
    2323    </ul> 
    2424</li> 
    25 <li><a href='modeling.html'>Application Developers: Using Dejavu to construct a model</a> 
     25<li><a href='modeling.html'>Application Designers: Using Dejavu to construct a model</a> 
    2626    <ul> 
    2727    <li>Units 
    …… 
    4545        </ul> 
    4646    </li> 
     47    <li>Associations between Unit Classes 
     48        <ul> 
     49        <li><tt>related_units</tt> methods</li> 
     50        <li><tt>Unit.first()</tt></li> 
     51        </ul> 
     52    </li> 
     53    <li>The Arena Object 
     54        <ul> 
     55        <li>Loading Stores</li> 
     56        <li>Registering Unit Classes</li> 
     57        </ul> 
     58    </li> 
     59    </ul> 
     60</li> 
     61<li><a href='managing.html'>Application Developers: Managing Units</a> 
     62    <ul> 
    4763    <li>Querying 
    4864        <ul> 
    …… 
    5470        <li>Using <tt>comparison</tt> to form Expressions</li> 
    5571        <li>Exporting the <tt>logic</tt> module</li> 
    56         </ul> 
    57     </li> 
    58     <li>Associations between Unit Classes 
    59         <ul> 
    60         <li><tt>related_units</tt> methods</li> 
    61         <li><tt>Unit.first()</tt></li> 
    6272        </ul> 
    6373    </li> 
    …… 
    7383        </ul> 
    7484    </li> 
    75     <li>The Arena Object 
    76         <ul> 
    77         <li>Loading Stores</li> 
    78         <li>Registering Unit Classes</li> 
    79         </ul> 
    80     </li> 
    8185    </ul> 
    8286</li> 
    …… 
    8791        <li>Database Storage Managers 
    8892            <ul> 
    89             <li>Microsoft Access (Jet)</li> 
    90             <li>Microsoft SQL Server</li> 
     93            <li>Microsoft SQL Server / Microsoft Access (Jet)</li> 
    9194            <li>PostgreSQL</li> 
    9295            <li>MySQL</li> 
    …… 
    120123            </ul> 
    121124        </li> 
    122         <li>Other Serialization Mechanisms</li> 
     125        <li>Other Serialization Mechanisms 
     126            <ul> 
     127            <li>sockets</li> 
     128            </ul> 
     129        </li> 
    123130        </ul> 
    124131    </li> 

  • trunk/doc/modeling.html

    r45 r46  
    1111<body> 
    1212 
    13 <h2>Application Developers: Using Dejavu to Construct a Domain Model</h2> 
     13<h2>Application Designers: Using Dejavu to Construct a Domain Model</h2> 
    1414 
    1515<h3>Units</h3> 
    …… 
    119119This is significant, because it allows us to store metadata about the 
    120120property itself: 
    121 <pre>>>> c.key, c.index, c.type, c.hints 
    122 ('ColorCopies', False, &lt;type 'bool'>, {})</pre> 
    123 The <tt>key</tt> attribute is merely the property's canonical name. The 
    124 <tt>index</tt> value tells Storage Managers whether or not to index the 
    125 column. The <tt>type</tt> attribute limits property values to instances 
    126 of that type (or <tt>None</tt>). Finally, the <tt>hints</tt> dictionary 
    127 provides hints to Storage Managers to help optimize storage. A common use, 
    128 for example, is to inform Managers that would usually store unicode strings 
    129 as strings of unlimited length, that a particular value should be a smaller 
    130 object; this is done with a 'bytes' mapping, such as <tt>hints = 
    131 {u'bytes': 255}</tt>. A bytes value of 0 implies no limit.</p> 
    132  
    133 <p>When you define a UnitProperty instance, you can pass in these extra 
     121<pre>>>> c.type, c.index, c.hints, c.key 
     122(&lt;type 'bool'>, False, {}, 'ColorCopies')</pre> 
     123 
     124When you define a UnitProperty instance, you can pass in these extra 
    134125attributes. The signature for UnitProperty is <tt>(type=unicode, 
    135 index=False, hints={}, key=None)</tt>. Supply any, all, or none of them 
    136 as needed.</p> 
     126index=False, hints={}, key=None)</tt>. Supply any, all, or none of them as 
     127needed. The <tt>key</tt> attribute is merely the property's canonical name, 
     128and is usually set for you. The <tt>index</tt> value tells database Storage 
     129Managers whether or not to index the column (if they do any indexing). The 
     130<tt>type</tt> attribute limits property values to instances of that type 
     131(or <tt>None</tt>). Finally, the <tt>hints</tt> dictionary provides hints 
     132to Storage Managers to help optimize storage. If you write a custom Storage 
     133Manager, you may define and use your own hints. Here are the ones that most 
     134builtin SM's understand:</p> 
     135 
     136<table> 
     137<tr><th>Key</th><th>Values</th><th>Description</th></tr> 
     138<tr> 
     139    <td>bytes</td> 
     140    <td>&gt;= 0</td> 
     141    <td>Inform SMs that would usually store unicode strings as strings of 
     142        unlimited length, that a particular value should be a smaller 
     143        object. A value of 0 implies no limit.</td> 
     144</tr> 
     145<tr> 
     146    <td>scale</td> 
     147    <td>&gt;= 0</td> 
     148    <td>Scale is the number of digits to the right of the decimal point 
     149        in a NUMERIC (fixedpoint or decimal) field. This hint informs SMs 
     150        that would usually store such data at a default scale (usually 2), 
     151        that the property should use a different scale.</td> 
     152</tr> 
     153<tr> 
     154    <td>precision</td> 
     155    <td>&gt;= 0</td> 
     156    <td>Precision is the total number of digits in a NUMERIC (fixedpoint or 
     157        decimal) field. This hint informs SMs that would usually store such 
     158        data at maximum precision, that the property should be a smaller 
     159        object. A value of 0 (or no hint) implies the maximum that the SM 
     160        can supply. PostgreSQL, for example, can handle 1000 digits. Note 
     161        that the <tt>fixedpoint</tt> module uses the word "precision" where 
     162        we use the word "scale"; it actually has unlimited precision (as  
     163        we use the word). The <tt>decimal</tt> module, in contrast, has 
     164        limited precision but no scale.</td> 
     165</tr> 
     166</table> 
     167 
    137168 
    138169<h4>Triggers</h4> 
    …… 
    254285expect to retrieve instances. The second argument should be an instance 
    255286of <tt>dejavu.logic.Expression</tt>, an object which encapsulates your 
    256 specific query (see <a href='#Querying'>Querying</a>). An example recall 
    257 operation: 
     287specific query (see <a href='managing.html#Querying'>Querying</a>). 
     288An example recall operation: 
    258289<pre>>>> e = logic.Expression(lambda x: x.Year == 1928) 
    259290>>> units = box.recall(Publication, e) 
    …… 
    338369function by passing attrs = ['ID']. Sandboxes provide a 
    339370<tt>count(cls, expr)</tt> method which does just this.</p> 
    340  
    341 <h3><a name='Querying'>Querying</a></h3> 
    342 <p>When you retrieve Units, you often don't want to load the entire set for 
    343 a given class. In Dejavu, you filter the set according to the UnitProperty 
    344 attributes for each object. Naturally, there must be a way to express 
    345 the filter you intend. Dejavu actually provides three ways, all in the 
    346 <tt>dejavu.logic</tt> module: <tt>Expression</tt>, 
    347 <tt>filter</tt>, and <tt>comparison</tt>.</p> 
    348  
    349 <h4>The <tt>Expression</tt> class</h4> 
    350 <p>Regardless of which technique you use to express your filter, you're 
    351 going to end up with a <tt>logic.Expression</tt> object. You can build 
    352 an Expression directly, passing a single lambda as an argument: 
    353 <pre>>>> from dejavu import logic 
    354 >>> import datetime 
    355 >>> f = lambda x: x.Date >= datetime.date(2004, 3, 1) 
    356 >>> e = logic.Expression(f) 
    357 >>> e 
    358 logic.Expression(lambda x: x.Date >= datetime.date(2004, 3, 1))</pre> 
    359 Neat, eh? I worked hard on that __repr__. ;)</p> 
    360  
    361 <p>It may be obvious, but we'll be explicit, here. The lambda which you pass 
    362 into an Expression must possess a single positional argument, which will 
    363 always be bound to a Unit instance. In the example above, it's named 'x', 
    364 but you can use any name you like. Using lambdas as a base means that we 
    365 can simply call <tt>Expression.evaluate(unit)</tt>, and receive a boolean 
    366 value indicating whether our Unit "passes the test". Attribute lookups on 
    367 our 'x' object will apply to Unit Properties for that Unit object. 
    368 That is, <tt>x.Date</tt> becomes <tt>unit.Date</tt>.</p> 
    369  
    370 <h4>Early binding</h4> 
    371 <p>What is not obvious from the above code snippet is perhaps the <b>most 
    372 important aspect</b> of Expressions: any globals or cell references (from  
    373 closures) in the supplied lambda get <b>bound early</b>. Compare the 
    374 following disassemblies: 
    375 <pre>>>> import dis 
    376 >>> dis.dis(f) 
    377   1           0 LOAD_FAST                0 (x) 
    378               3 LOAD_ATTR                1 (Date) 
    379               6 LOAD_GLOBAL              2 (datetime) 
    380               9 LOAD_ATTR                3 (date) 
    381              12 LOAD_CONST               1 (2004) 
    382              15 LOAD_CONST               2 (3) 
    383              18 LOAD_CONST               3 (1) 
    384              21 CALL_FUNCTION            3 
    385              24 COMPARE_OP               5 (>=) 
    386              27 RETURN_VALUE         
    387 >>> dis.dis(e.func) 
    388   1           0 LOAD_FAST                0 (x) 
    389               3 LOAD_ATTR                1 (Date) 
    390               6 LOAD_CONST               6 (datetime.date(2004, 3, 1)) 
    391               9 COMPARE_OP               5 (>=) 
    392              12 RETURN_VALUE         
    393 </pre> 
    394 As you can see, the function itself references the global 'datetime' module. 
    395 Once we wrap it in the Expression, however, it becomes a constant! Thanks to 
    396 Raymond Hettinger for inspiring this solution <a href='#hettinger'>[1]</a>. 
    397 Early binding, however, implies two consequences:</p> 
    398  
    399 <p>First, any globals or cell references must be present in the lambda's 
    400 scope when it is passed into Expression(). This is the norm and shouldn't 
    401 require too much thought from you when you write Expressions. In the 
    402 example above, we simply imported <tt>datetime</tt> as you would expect.</p> 
    403  
    404 <p>Second, any globals or cell references must <b>also</b> be present in 
    405 the <tt>logic</tt> module's globals when the Expression is unpickled. 
    406 Pickling occurs when Expressions are sent over sockets, and also if 
    407 Expressions are themselves persisted to storage (for example, see 
    408 <u>Unit Engines</u>, below). This means your application should inject 
    409 globals into the <tt>logic</tt> module. Note that the <tt>logic</tt> module 
    410 already tries to import <tt>datetime</tt>, <tt>fixedpoint</tt> and 
    411 <tt>decimal</tt>.</p> 
    412  
    413 <h4>External functions within Expressions</h4> 
    414 <p>Dejavu provides additional functions which can be used in Expressions. 
    415 For example, you can construct an Expression like: 
    416 <pre>logic.Expression(lambda x: x.Size < 3 and x.Date > dejavu.today())</pre> 
    417 In this example, the <tt>today()</tt> function breaks convention and is 
    418 actually <b>bound late</b>. That is, if you construct this Expression now 
    419 and use it six months later, the value of <tt>today()</tt> will change. 
    420 Storage Managers "know about" these dejavu functions, and can use them 
    421 to build more appropriate queries. Here are the functions supplied by 
    422 the <tt>dejavu</tt> module:</p> 
    423  
    424 <table> 
    425 <tr><th>Function</th><th>Late bound?</th><th>Description</th></tr> 
    426 <tr> 
    427     <td><tt>icontains(a, b)</tt></td> 
    428     <td></td> 
    429     <td>Case-insensitive test b in a. Note the operand order.</td> 
    430 </tr> 
    431 <tr> 
    432     <td><tt>icontainedby(a, b)</tt></td> 
    433     <td></td> 
    434     <td>Case-insensitive test a in b. Note the operand order.</td> 
    435 </tr> 
    436 <tr> 
    437     <td><tt>istartswith(a, b)</tt></td> 
    438     <td></td> 
    439     <td>True if a starts with b (case-insensitive), False otherwise.</td> 
    440 </tr> 
    441 <tr> 
    442     <td><tt>iendswith(a, b)</tt></td> 
    443     <td></td> 
    444     <td>True if a ends with b (case-insensitive), False otherwise.</td> 
    445 </tr> 
    446 <tr> 
    447     <td><tt>ieq(a, b)</tt></td> 
    448     <td></td> 
    449     <td>True if a == b (case-insensitive), False otherwise.</td> 
    450 </tr> 
    451 <tr> 
    452     <td><tt>year(value)</tt></td> 
    453     <td></td> 
    454     <td>The year attribute of a date. If value is None, return None.</td> 
    455 </tr> 
    456 <tr> 
    457     <td><tt>now()</tt></td> 
    458     <td>Y</td> 
    459     <td>datetime.datetime.now()</td> 
    460 </tr> 
    461 <tr> 
    462     <td><tt>today()</tt></td> 
    463     <td>Y</td> 
    464     <td>datetime.date.today()</td> 
    465 </tr> 
    466 <tr> 
    467     <td><tt>iscurrentweek(value)</tt></td> 
    468     <td>Y</td> 
    469     <td>If value is in the current week, return True, else False.</td> 
    470 </tr> 
    471 </table> 
    472  
    473 <p>It is possible for you, the application developer, to define your 
    474 own external functions. However, because Storage Managers are unaware 
    475 of your new functions, they will not be able to optimize their use; 
    476 instead, they will simply retrieve a larger set of objects from storage, 
    477 evaluate each one against the function you provide, and return those 
    478 Units which match your function. This isn't necessarily a bad thing; 
    479 it provides the same functionality as if you wrote the test inline 
    480 within your own code. By making that test a logic function, you allow 
    481 it to be stored in Engine <i>rules</i> (see <u>Unit Engines</u>,  
    482 below).</p> 
    483  
    484 <h4>Combining Expressions</h4> 
    485 <p>Expressions are combinable; by using the <tt>&</tt> operator, the two 
    486 expressions are combined with an adjoining logical "and". For example: 
    487 <pre>>>> a = logic.Expression(lambda x: x.Size > 3) 
    488 >>> b = logic.Expression(lambda x: x.Size <= 15) 
    489 >>> c = a & b 
    490 >>> c 
    491 logic.Expression(lambda x: (x.Size > 3) and (x.Size <= 15))</pre> 
    492 The <tt>+</tt> operator works just like the <tt>&</tt> operator. The 
    493 <tt>|</tt> operator combines the two Expressions with a logical 'or'.</p> 
    494  
    495 <h4>Using <tt>filter</tt> to form Expressions</h4> 
    496 <p>The <tt>logic</tt> module also provides convenient methods to 
    497 create common types of Expression objects via the <tt>filter</tt> and 
    498 <tt>comparison</tt> factory functions.</p> 
    499  
    500 <p>The <tt>filter(**kwargs)</tt> function produces an Expression by taking 
    501 the keyword arguments you supply, and rewriting them in lambda form. The  
    502 only operator allowed is therefore the equals '==' operator. For example: 
    503 <pre>>>> logic.filter(Type='Cat', Mutation='Atomic') 
    504 logic.Expression(lambda x: (x.Type == 'Cat') and (x.Mutation == 'Atomic'))</pre> 
    505 </p> 
    506  
    507 <h4>Using <tt>comparison</tt> to form Expressions</h4> 
    508 <p>The <tt>comparison(attr, cmp_op, criteria)</tt> function allows you to 
    509 form Expressions with dynamic operators. This can come in handy when you 
    510 are constructing Expressions on the fly from user input. For example, a 
    511 search page might prompt users for an attribute name, an operator, and an 
    512 operand (the criteria).</p> 
    513  
    514 <p>Borrowing from <tt>opcode.cmp_op</tt>, the allowed values for our cmp_op 
    515 argument are as follows:</p> 
    516 <table> 
    517 <tr><th>Numeric Value (cmp_op)</th><th>Operator</th></tr> 
    518 <tr><td>0</td><td>&lt;</td></tr> 
    519 <tr><td>1</td><td>&lt;=</td></tr> 
    520 <tr><td>2</td><td>==</td></tr> 
    521 <tr><td>3</td><td>!=</td></tr> 
    522 <tr><td>4</td><td>&gt;</td></tr> 
    523 <tr><td>5</td><td>&gt;=</td></tr> 
    524 <tr><td>6</td><td>in</td></tr> 
    525 <tr><td>7</td><td>not in</td></tr> 
    526 <tr><th colspan='2'>Most SM's don't support the following:</th></tr> 
    527 <tr><td>8</td><td>is</td></tr> 
    528 <tr><td>9</td><td>is not</td></tr> 
    529 </table> 
    530  
    531 <p>Here's an example of using <tt>comparison</tt>: 
    532 <pre>>>> logic.comparison('Name', 3, 'Mr. Kamikaze') 
    533 logic.Expression(lambda x: x.Name != 'Mr. Kamikaze')</pre> 
    534 Although the comparison function only allows a single comparison at a time, 
    535 the resulting Expressions can be combined with the <tt>&</tt> and <tt>|</tt> 
    536 operators (described earlier) to produce more complex Expressions.</p> 
    537  
    538 <h4>Exporting the <tt>logic</tt> module</h4> 
    539 <p>The <tt>logic</tt> module (and <tt>codewalk</tt>, on which it is built) 
    540 isn't limited to Dejavu. Feel free to use it in some other framework or 
    541 script! The only change you may have to make (if you relocate the module 
    542 outside of the <tt>dejavu</tt> package) would be to the single line: 
    543 <tt>from dejavu import codewalk</tt>, to point to the new location.</p> 
    544  
    545 <p>In particular, <tt>logic.Expression</tt> objects can operate on <i>any</i> 
    546 Python object, not just dejavu <tt>Unit</tt> instances. If you wish to 
    547 provide additional logic functions (as dejavu does), simply inject them 
    548 into <tt>logic</tt>'s globals.</p> 
    549  
    550 <p>You may also find the underlying <tt>codewalk</tt> module useful for 
    551 other purposes on its own. The <tt>Visitor</tt> base class can be very 
    552 convenient for building bytecode hacks.</p> 
    553  
    554 <p>To make a long story short, Dejavu depends on <tt>logic</tt> throughout, 
    555 but the reverse is not true.</p> 
    556371 
    557372 
    …… 
    639454such far Unit or None if not found.</p> 
    640455 
    641 <h3>Unit Engines</h3> 
    642 <p>Once you've created and associated your Unit classes, you can begin to 
    643 write "business logic" code (mostly inside those classes, we hope), and 
    644 "presentation logic" code (mostly outside those classes). In most cases, 
    645 you will construct Expressions within your own code manually to retrieve 
    646 Units. Sometimes, however, you need to persist query parameters from your 
    647 users; in other cases, you might store a list of Units which match a query 
    648 (regardless of who formed the necessary Expression). Finally, you might 
    649 wish to manipulate lists of Units as sets: differences, intersections, 
    650 and unions. The <tt>engines</tt> module addresses all of these needs.</p> 
    651  
    652 <h4>Collections: Lists of Units</h4> 
    653 <p>The <tt>UnitCollection</tt> class provides a means of storing a list 
    654 of Units, or rather, a list of Unit ID's. You use its <tt>Type</tt> 
    655 property to indicate the class of the indexed Units. That value should be 
    656 the <b>name</b> of the Unit Class, <b>not</b> the class object itself 
    657 (this is different than most other calls in Dejavu). If you need to 
    658 retrieve the actual Unit class, call <tt>UnitCollection().unit_class()</tt>.</p> 
    659  
    660 <p><tt>UnitCollection</tt> itself subclasses <tt>dejavu.Unit</tt>; you can 
    661 therefore persist Unit Collections via Dejavu Storage Managers (most SM's, 
    662 anyway; it's recommended that SM's handle Unit Collections, but not 
    663 required. Check your SM to see if it does).</p> 
    664  
    665 <p>Each Collection has a thread lock (an RLock, actually) which you should 
    666 <tt>acquire()</tt> before you add an ID to the set, and <tt>release()</tt> 
    667 afterward. If you use the <tt>add(ID)</tt> method, this locking is done 
    668 for you.</p> 
    669  
    670 <p>When you need to retrieve the actual Units which are indexed by the 
    671 Collection, call the <tt>units(quota=None)</tt> method, which will 
    672 look up the Units and return them in a list. Since the Collection only 
    673 stores ID's, it is possible that one of the indexed Units may have been 
    674 destroyed since the list was built. The <tt>units</tt> method simply 
    675 passes over these "phantom" Units. You can inspect the full list of IDs 
    676 in the Collection (whether they reference existing Units or not) with 
    677 the <tt>ids()</tt> method.</p> 
    678  
    679 <p>Collections also provide a convenience function for grouping Units 
    680 by attribute: <tt>xdict(attr)</tt>. This function will look up each Unit 
    681 in the Collection, inspect the attribute that you specify, and return 
    682 a dictionary of the form <tt>{attr_val1: [Unit, Unit, ...]}</tt>. 
    683 Each distinct attribute value will have its own key, with a list of 
    684 matching Units as the value.</p> 
    685  
    686 <h4>Engines</h4> 
    687 <p>You can form Collections by hand, but a more powerful technique is 
    688 the <tt>UnitEngine</tt>, a factory for Collections. Engines are very 
    689 simple: they possess a set of <i>rules</i> which are executed when 
    690 you want to take a <i>snapshot</i> of Units. The snapshot which is 
    691 produced is a <tt>UnitCollection</tt> object. Whenever you call 
    692 <tt>take_snapshot()</tt>, the Engine will maintain an association 
    693 to the resulting Collection. You can access past snapshots with the 
    694 <tt>snapshots()</tt> method.</p> 
    695  
    696 <p>Engines are themselves Units, and can be persisted via Storage Managers. 
    697 The only properties they possess are: an <tt>ID</tt>, a <tt>Name</tt>, 
    698 an <tt>Owner</tt>, a <tt>FinalClassName</tt>, and <tt>Created</tt>, 
    699 the creation date of the Engine.</p> 
    700  
    701 <p>The <tt>Owner</tt> property should either be a user name, or one of the 
    702 reserved names: "Public" and "System". By default, the <tt>permit()</tt> 
    703 method allows a user read-access to the Engine if they are the Owner, or 
    704 the Owner is "Public" or "System". Write-access is permitted if the user 
    705 is the Owner, or the Owner is "Public". Feel free to override 
    706 <tt>permit()</tt> in a subclass to provide different behaviors.</p> 
    707  
    708 <p>The <tt>FinalClassName</tt> is set for you as you add Rules to the 
    709 Engine. You can use the value of this property, for example, to tell 
    710 your users, "Engine #23569 is an 'Armadillo' engine," when it produces 
    711 Collections of <tt>Armadillo</tt> Units. The only time you might want to 
    712 set this value is when you first create the Engine, before you have added 
    713 any Rules.</p> 
    714  
    715 <h4>Rules</h4> 
    716 <p>Just like Collections and Engines, <tt>UnitEngineRule</tt> is <i>also</i> 
    717 a subclass of <tt>Unit</tt>, and can be persisted via Storage Managers. All 
    718 three work together to provide a complete, dynamic, application-level query 
    719 generator.</p> 
    720  
    721 <p>Okay, so what are Rules? You might say they're a "little language", 
    722 with the following primitives, or "operations":</p> 
    723 <table> 
    724 <tr><th>Operation</th><th>Operand(s)</th><th>Description</th></tr> 
    725 <tr><th colspan='3'>Operations on a single set</th></tr> 
    726 <tr> 
    727     <td>CREATE</td> 
    728     <td>The classname of the new Type</td> 
    729     <td>Creates a new Set of the specified Type. All Units of that Type 
    730         are included in the new Set.</td> 
    731 </tr> 
    732 <tr> 
    733     <td>FILTER</td> 
    734     <td>A <tt>logic.Expression</tt></td> 
    735     <td>Removes Units from the current Set which do not match the 
    736         Expression.</td> 
    737 </tr> 
    738 <tr> 
    739     <td>FUNCTION</td> 
    740     <td>The name of a function in the <tt>Arena.engine_functions</tt> 
    741         dict</td> 
    742     <td>Calls the function, passing the current Set. The function 
    743         should modify the Set.</td> 
    744 </tr> 
    745 <tr> 
    746     <td>TRANSFORM</td> 
    747     <td>The classname of the new Type</td> 
    748     <td>Transform the current Set into a Set of associated Units 
    749         (of another Type). The association must be present in the 
    750         <tt>Arena.associations</tt> graph.</td> 
    751 </tr> 
    752 <tr> 
    753     <td>RETURN</td> 
    754     <td></td> 
    755     <td>Optional. If omitted, the last Set handled is returned as the 
    756         snapshot. If supplied, the ID of the Set to return.</td> 
    757 </tr> 
    758 <tr><th colspan='3'>Operations on two sets</th></tr> 
    759 <tr> 
    760     <td>COPY</td> 
    761     <td>The Set ID of the new Set</td> 
    762     <td>Copies the current Set to a new Set. The current Set is unchanged.</td> 
    763 </tr> 
    764 <tr> 
    765     <td>DIFFERENCE</td> 
    766     <td>The ID of the Set to mix in</td> 
    767     <td>Removes IDs from the current Set which exist in the second Set.</td> 
    768 </tr> 
    769 <tr> 
    770     <td>INTERSECTION</td> 
    771     <td>The ID of the Set to mix in</td> 
    772     <td>Removes IDs from the current Set which <i>do not</i> exist in the 
    773         second Set.</td> 
    774 </tr> 
    775 <tr> 
    776     <td>UNION</td> 
    777     <td>The ID of the Set to mix in</td> 
    778     <td>Adds any IDs to the current Set which exist in the second Set.</td> 
    779 </tr> 
    780 </table> 
    781  
    782 <p>Each Rule has an <tt>Operation</tt> property (a string, one of the above), 
    783 a <tt>SetID</tt>, and an <tt>Operand</tt>. Here's an example ruleset:</p> 
    784 <table> 
    785 <tr><th>Sequence</th><th>Operation</th><th>SetID</th><th>Operand</th></tr> 
    786 <tr><td>1</td><td>CREATE</td><td>1</td><td>Invoice</td></tr> 
    787 <tr><td>2</td><td>FILTER</td><td>1</td><td>(Expression)</td></tr> 
    788 <tr><td>3</td><td>CREATE</td><td>2</td><td>Inventory</td></tr> 
    789 <tr><td>4</td><td>FILTER</td><td>2</td><td>(Expression)</td></tr> 
    790 <tr><td>5</td><td>TRANSFORM</td><td>2</td><td>Invoice</td></tr> 
    791 <tr><td>6</td><td>DIFFERENCE</td><td>1</td><td>2</td></tr> 
    792 <tr><td>7</td><td>RETURN</td><td>1</td><td></td></tr> 
    793 </table> 
    794  
    795 <p>As you can see, every Rule operates on a <i>Set</i> of Units. The first 
    796 rule is always to CREATE a set, declaring it to contain a certain Type 
    797 of Units. In most cases, you will then FILTER that set. If you simply 
    798 created a set and then returned it, it would contain all Units of the 
    799 declared Type. When you filter a set, however, you remove Units from 
    800 the whole which do not match the filter's Expression.</p> 
    801  
    802 <p>In the example above, we CREATE a second Set so that we can eventually 
    803 obtain the DIFFERENCE between Set 1 and Set 2. The second Set contains 
    804 Units of a different Type than the first. Once we filter Set 2, we then 
    805 TRANSFORM it; for each Inventory Unit, we look up associated Invoice 
    806 Units. Then, we find the difference between the two Invoice sets and 
    807 RETURN it.</p> 
    808  
    809 <p>Rules are executed in order according to their <tt>Sequence</tt> 
    810 attribute (lowest first). When you use the <tt>Engine.add_rule</tt> method, 
    811 the next <tt>Sequence</tt> value is retrieved for you. Notice that each 
    812 Rule belongs to one and only one Engine; they are not shared between 
    813 Engines. Each Rule has its own <tt>EngineID</tt> attribute.</p> 
    814  
    815 <h4>Engine Functions</h4> 
    816 <p>The FUNCTION rule deserves special mention. The Operand of a FUNCTION 
    817 rule is a string, a key in the <tt>Arena.engine_functions</tt> dictionary. 
    818 When the rule is executed, that key is used to look up the function, which 
    819 is then called, passing <tt>(sandbox, set)</tt>. The function should 
    820 mutate the set directly. Use FUNCTION rules to mutate sets in ways which 
    821 are more complex than those provided by FILTER and TRANSFORM. For example, 
    822 you might provide a function which removes all but the first Unit in the 
    823 Set (according to some ordering algorithm).</p> 
    824  
    825  
    826 <h3>Analysis Tools</h3> 
    827 <p>Dejavu includes various tools to help you manipulate groups of Units.</p> 
    828  
    829 <h4>Sorting Units</h4> 
    830 <p>When you recall Units, you receive a generator, and must iterate over 
    831 the values in some way. Often, this is accomplished with a list 
    832 comprehension: 
    833 <pre>f = logic.Expression(lambda x: 'Aa' in x.Name) 
    834 people = [x for x in sandbox.recall(Person, f)] 
    835 </pre> 
    836 However, the <tt>recall</tt> method doesn't do any sorting; you must sort 
    837 your list in your Python code. Dejavu provides a <tt>sort(attrs, 
    838 descending=False)</tt> function to assist you. It returns a function, which 
    839 you can then use in Python's sort function. Continuing our example: 
    840 <pre>sorted_people = people.sort(dejavu.sort('Size', 'Name'))</pre> 
    841 The most important issue (and the reason we don't just use 2.4's attrgetter), 
    842 is that any Unit property must allow values of None, which tends to raise 
    843 errors when compared to values of other types. The function which 
    844 <tt>sort</tt> creates for you treats None as "less than" any other value.</p> 
    845  
    846 <h4>Cross-tabulation</h4> 
    847 <p>Cross-tabs (also called <i>aggregate tables</i> or <i>pivot tables</i>) 
    848 display aggregate information about objects by category. For example, 
    849 rather than show a list of Safari records, one row per trip, you might 
    850 wish to show a table where each row represents a Destination, and each 
    851 column shows the count of Safaris to that Destination for each distinct 
    852 Year. In this example, we say that the Safaris are "grouped by" their 
    853 Destination values, and that we "pivot" on the Year values.</p> 
    854  
    855 <p>Dejavu helps you form such a table via the <tt>CrossTab</tt> class. 
    856 You need to specify the group(s) you wish to use, and the pivot attribute. 
    857 Finally, you must specify the aggregate function. Here's a code example: 
    858 <pre> 
    859 >>> data = ["a", "b", "cc", "bddd", "a4", "b6"] 
    860 >>> group = lambda x: x.isalpha() 
    861 >>> pivot = lambda x: x[0] 
    862 >>> ctab = analysis.CrossTab(data, [group], pivot, dejavu.COUNT) 
    863 >>> data, columns = ctab.results() 
    864 >>> data 
    865 {(True,): {"a": 1, "b": 2, "c": 1}, 
    866  (False,): {"a": 1, "b": 1}} 
    867 >>> columns 
    868 ["a", "b", "c"]</pre> 
    869 You may notice that we're not using Units in our example; the 
    870 <tt>CrossTab</tt> class is designed to work with any objects. Here's one 
    871 way to lay out that data:</p> 
    872 <table> 
    873 <tr><th>Is Alpha</th><th>a</th><th>b</th><th>c</th></tr> 
    874 <tr><td>Y</td><td>1</td><td>2</td><td>1</td></tr> 
    875 <tr><td>N</td><td>1</td><td>1</td><td>0</td></tr> 
    876 </table> 
    877  
    878 <p>The <tt>results</tt> method returns two values. First, the table 
    879 itself in the form of a dictionary; each key is a tuple of group values, 
    880 and the corresponding value is a sub-dictionary. Each sub-dict has keys 
    881 which are the pivot attribute, and values which equal the aggregates. 
    882 I know, that was confusing; look at the example. The second value to 
    883 be returned is a list of the pivot column values; you'll notice they're 
    884 sorted.</p> 
    885  
    886 <p>The groups and pivot arguments may be either strings or functions. 
    887 If strings, they must be the names of attributes of the source objects. 
    888 The final aggfunc argument defaults to COUNT, but may also be SUM. 
    889 More aggfuncs may arrive in the future.</p> 
    890456 
    891457<h3>The Arena Object</h3> 
    …… 
    925491<hr /> 
    926492 
    927 <p><a name='hettinger'>[1]</a> Python Cookbook, 
    928 <a href='http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/277940'>Binding 
    929 Constants at compile time</a><br /> 
    930 </p> 
    931  
    932493</body> 
    933494</html> 

  • trunk/doc/storage.html

    r45 r46  
    4040 
    4141<p>The third line in our example is a standard ODBC-style Connect string. 
    42 The _MSAccess class requires this entry; other SM's may not. There are very 
    43 few configuration entries which apply to all Storage Managers. Here are 
    44 the common ones which do:</p> 
     42The _MSAccess class requires this entry; other SM's may not. There are a few 
     43configuration entries which (probably) apply to all Storage Managers:</p> 
    4544 
    4645<table> 
    …… 
    6665    <td>Units</td> 
    6766    <td><tt>[UnitCollection, UnitEngine, UnitEngineRule, FieldDashboardSumSet]</tt></td> 
    68     <td>Optional. Declares which Unit classes to manage with this SM.</td> 
     67    <td>Optional. Declares which Unit classes to manage with this SM 
     68        (see below).</td> 
    6969</tr> 
    7070<tr> 
    …… 
    8585    <td>Create If Missing</td> 
    8686    <td><tt>True</tt></td> 
    87     <td>Optional. If not blank or missing, create the database as needed.</td> 
     87    <td>Optional. If not blank or missing, create the database as needed. 
     88        Because of the vagaries of various databases, the ODBC Storage 
     89        Manager doesn't support this.</td> 
     90</tr> 
     91<tr> 
     92    <td>Expanded Columns</td> 
     93    <td><tt>Animal.PreviousZoos:int, Exhibit.Animals:int</tt></td> 
     94    <td>Optional. A comma-separated list of UnitClass.Property:subtype 
     95        strings. Each such property should be of .type list or tuple. 
     96        Usually, lists are pickled for storage in a normal database 
     97        field. Properties listed in <tt>Expanded Columns</tt> will 
     98        be stored each in their own table. The "subtype" portion tells 
     99        the Storage Manager the type of each value in the list.</td> 
    88100</tr> 
    89101</table> 
    …… 
    100112<h4>Database Storage Managers</h4> 
    101113 
    102 <h5>Microsoft Access (Jet)</h5> 
    103 <p>This class was developed against ADO 2.7. Configuration entries:</p> 
    104 <ul> 
    105     <li><b>Class:</b> <tt>dejavu.storage.storeado.StorageManagerADO_MSAccess</tt></li> 
     114<h5>Microsoft SQL Server / Microsoft Access (Jet)</h5> 
     115<p>This module was developed against ADO 2.7. Configuration entries:</p> 
     116<ul> 
     117    <li><b>Class:</b> <tt>dejavu.storage.storeado.StorageManagerADO_SQLServer</tt> 
     118        or <tt>dejavu.storage.storeado.StorageManagerADO_MSAccess</tt></li> 
    106119    <li><b>Connect:</b> A valid ADO connect string. There are plenty of 
    107120        online references for how to form these; for example, at 
    …… 
    115128</ul> 
    116129 
    117 <h5>Microsoft SQL Server</h5> 
    118 <p>This class was developed against ADO 2.7. Configuration entries:</p> 
    119 <ul> 
    120     <li><b>Class:</b> <tt>dejavu.storage.storeado.StorageManagerADO_SQLServer</tt></li> 
    121     <li><b>Connect:</b> A valid ADO connect string. There are plenty of 
    122         online references for how to form these; for example, at 
    123         <a href='http://support.microsoft.com/?kbid=193332'>Microsoft</a>.</li> 
    124     <li><b>CursorType:</b> Optional. Passed to Recordset.Open(). See 
    125         <a href='http://msdn.microsoft.com/library/en-us/ado270/htm/mdmthrstopen.asp'>Microsoft</a> 
    126         again.</li> 
    127     <li><b>LockType:</b> Optional. Passed to Recordset.Open(). See 
    128         <a href='http://msdn.microsoft.com/library/en-us/ado270/htm/mdmthrstopen.asp'>Microsoft</a> 
    129         again.</li> 
    130 </ul> 
    131  
    132130<h5>PostgreSQL (pyPgSQL)</h5> 
    133131<p>This class was developed against PostgreSQL 8.0.0 rc-1. Configuration entries:</p> 
    …… 
    160158    <li><b>Database:</b> Filename of the database. May be a relative path. 
    161159        If the DB does not already exist, it will be created.</li> 
     160    <li><b>Mode:</b> Optional. DB file mode. Defaults to 0755.</li> 
    162161</ul> 
    163162 
    164163<h5>ODBC</h5> 
    165 <p>This class doesn't currently work. It needs some updating. Configuration 
     164<p>This class doesn't support create_database or drop_database. Configuration 
    166165entries:</p> 
    167166<ul> 
    168167    <li><b>Class:</b> <tt>dejavu.storage.storeodbc.StorageManagerODBC</tt></li> 
    169     <li><b>Connect:</b> A valid ADO connect string. There are plenty of 
    170         online references for how to form these; for example, at 
    171         <a href='http://support.microsoft.com/?kbid=193332'>Microsoft</a>.</li> 
     168    <li><b>Connect:</b> A valid ODBC connect string.</li> 
    172169</ul> 
    173170 

  • trunk/storage/db.py

    r45 r46  
    11"""Base classes and tools for writing database Storage Managers. 
    22 
    3 Python -> [SQL repr of DB type ->] DB value -> Python 
     3Unit type -> [SQL repr ->] DB -> incoming Python value -> Unit type 
    44 
    55 
    66DATA TYPES 
    77========== 
    8 Since Dejavu relies on external database servers for its persistence, many 
     8Since Dejavu relies on external database servers for its persistence, 
    99Python datatypes must be converted to column types in the DB. When writing 
    1010a StorageManager, you should make sure that your type conversions can handle 
    1111at least the following limitations. If possible, implement the type with no 
    1212limits. Also, follow UnitProperty.hints['bytes'] where possible. A value 
    13 of zero for hints['bytes'] implies no limit. If no value is given, you may 
    14 choose whatever default size you wish (255 is common for strings). 
     13of zero for hints['bytes'] implies no limit. If no value is given, try to 
     14assume no limit, although you may choose whatever default size you wish 
     15(255 is common for strings). 
    1516""" 
    1617 
    1718from types import FunctionType 
    18 from dejavu import codewalk 
     19import dejavu 
     20from dejavu import codewalk, logic, storage 
    1921 
    2022try: 
    …… 
    3032    pass 
    3133 
     34try: 
     35    # Builtin in Python 2.5? 
     36    decimal 
     37except NameError: 
     38    try: 
     39        # Module in Python 2.3, 2.4 
     40        import decimal 
     41    except ImportError: 
     42        pass 
     43 
     44import warnings 
     45import threading 
     46 
    3247 
    3348class FieldTypeAdapter(object): 
    …… 
    3651    This base class is designed to work out-of-the-box with PostgreSQL 8. 
    3752    """ 
     53     
     54    # 1000 is the max precision for NUMERIC columns for PostgreSQL 8. 
     55    # Override in subclasses. 
     56    numeric_max_precision = 1000 
    3857     
    3958    def coerce(self, cls, key): 
    …… 
    90109    def coerce_datetime_time(self, cls, key): return u"TIME" 
    91110     
    92     # I was seriously disinterested in writing a parser for pg's interval. 
     111    # I was seriously disinterested in writing a parser for interval. 
    93112    def coerce_datetime_timedelta(self, cls, key): 
    94113        return self.coerce_float(cls, key) 
    95114     
     115    def coerce_decimal_Decimal(self, cls, key): 
     116        prop = getattr(cls, key) 
     117        precision = int(prop.hints.get('precision', '0')) 
     118        if precision == 0: 
     119            precision = decimal.getcontext().prec 
     120        if precision > self.numeric_max_precision: 
     121            warnings.warn("Decimal precision %s > maximum %s for %s.%s, " 
     122                          "using %s. Values may be stored incorrectly." 
     123                          % (precision, self.numeric_max_precision, 
     124                             cls.__name__, key, self.__class__.__name__)) 
     125            precision = self.numeric_max_precision 
     126        # Assume most people use decimal for money; default scale = 2. 
     127        scale = int(prop.hints.get(u'scale', 2)) 
     128        return u"NUMERIC(%s, %s)" % (precision, scale) 
     129     
    96130    def coerce_decimal(self, cls, key): 
    97         # Notice that we do not specify precision or scale. In PostgreSQL, 
    98         # this means "numeric values of any precision and scale can be  
    99         # stored, up to the implementation limit on precision". 
    100         # We could inspect the 'bytes' hint and try to convert that to 
    101         # precision, but... meh. That's why they're called "hints". 
    102         return u"NUMERIC" 
    103     coerce_decimal_Decimal = coerce_decimal 
     131        # If decimal ever becomes a builtin. Python 2.5? 
     132        return self.coerce_decimal_Decimal(cls, key) 
    104133     
    105134    def coerce_fixedpoint_FixedPoint(self, cls, key): 
    106         return self.coerce_decimal(cls, key) 
     135        prop = getattr(cls, key) 
     136        precision = int(prop.hints.get('precision', '0')) 
     137        if precision == 0: 
     138            precision = self.numeric_max_precision 
     139        # Assume most people use decimal for money; default scale = 2. 
     140        scale = int(prop.hints.get(u'scale', 2)) 
     141        return u"NUMERIC(%s, %s)" % (precision, scale) 
    107142     
    108143    def coerce_long(self, cls, key): 
    109144        prop = getattr(cls, key) 
    110         bytes = int(prop.hints.get(u'bytes', '0')) 
     145        bytes = int(prop.hints.get(u'bytes', 0)) 
    111146        if bytes <= 4: 
    112147            return self.coerce_int(cls, key) 
    …… 
    114149            # BIGINT is usually 8 bytes 
    115150            return "BIGINT" 
    116         # Anything larger than 8 bytes, use decimal/numeric 
    117         return "NUMERIC" 
     151        # Anything larger than 8 bytes, use decimal/numeric. 
     152        if bytes > self.numeric_max_precision: 
     153            warnings.warn("Long bytes %s > maximum %s for %s.%s, " 
     154                          "using %s. Values may be stored incorrectly." 
     155                          % (bytes, self.numeric_max_precision, 
     156                             cls.__name__, key, self.__class__.__name__)) 
     157            bytes = self.numeric_max_precision 
     158        return "NUMERIC(%s, 0)" % bytes 
    118159     
    119160    def coerce_int(self, cls, key): 
    …… 
    137178    """ 
    138179     
    139     # Notice these are ordered pairs. 
     180    # Notice these are ordered pairs. Escape \ before introducing new ones. 
    140181    escapes = [("'", "''"), ("\\", r"\\"), ("%", r"\%"), ("_", r"\_")] 
    141182     
    …… 
    169210        return 'FALSE' 
    170211     
     212    # The great thing about these 3 date coercers is that you can use 
     213    # them with (VAR)CHAR columns just as well as with DATETIME, etc. 
     214    # and comparisons will still work! 
    171215    def coerce_datetime_datetime(self, value): 
    172216        return (u"'%04d-%02d-%02d %02d:%02d:%02d'" % 
    …… 
    184228        return repr(float_val) 
    185229     
     230    coerce_decimal = tostr 
     231    coerce_decimal_Decimal = tostr 
    186232    coerce_fixedpoint_FixedPoint = tostr 
    187233    coerce_float = tostr 
    …… 
    214260    This base class is designed to work out-of-the-box with PostgreSQL 8. 
    215261    """ 
    216      
    217     def __init__(self, unit=None): 
    218         self.unit = unit 
    219262     
    220263    def coerce(self, value, coltype, valuetype=None): 
    …… 
    239282        return xform(value, coltype) 
    240283     
    241     def consume(self, key, value, coltype): 
    242         expectedType = self.unit.__class__.property_type(key) 
     284    def consume(self, unit, key, value, coltype): 
     285        expectedType = unit.__class__.property_type(key) 
    243286        value = self.coerce(value, coltype, expectedType) 
    244         # Set the attribute directly to avoid __set__ overhead. 
    245         self.unit._properties[key] = value 
     287        unit._properties[key] = value 
    246288     
    247289    def pickle(self, value, coltype): 
    …… 
    269311        days, seconds = divmod(value, 1) 
    270312        return datetime.timedelta(days, int(seconds * 86400)) 
     313     
     314    def coerce_decimal(self, value, coltype): 
     315        return decimal(str(value)) 
     316     
     317    def coerce_decimal_Decimal(self, value, coltype): 
     318        return decimal.Decimal(str(value)) 
    271319     
    272320    coerce_dict = pickle 
    …… 
    332380 
    333381class SQLDecompiler(codewalk.LambdaDecompiler): 
    334     """SQLDecompiler(tablename, expr, adapter=AdapterToSQL). 
     382    """SQLDecompiler(tablename, expr, adapter=AdapterToSQL()). 
    335383     
    336384    Produce SQL from a supplied Expression object, with a lambda of the form: 
    …… 
    351399    sql_cmp_op = ('<', '<=', '=', '!=', '>', '>=', 'in', 'not in') 
    352400     
    353     def __init__(self, tablename, expr, adapter=AdapterToSQL): 
     401    def __init__(self, tablename, expr, adapter=AdapterToSQL()): 
    354402        self.tablename = tablename 
    355403        self.expr = expr 
    356         self.adapter = adapter() 
     404        self.adapter = adapter 
    357405        obj = expr.func 
    358406        codewalk.LambdaDecompiler.__init__(self, obj) 
    …… 
    516564    def column_name(self, name): 
    517565        # This is valid SQL for PostgreSQL only and should be overridden. 
    518         return '"%s"."%s"' % (self.tablename, name) 
     566        return '%s."%s"' % (self.tablename, name) 
    519567     
    520568    # --------------------------- Dispatchees --------------------------- # 
    …… 
    571619        return "LENGTH(" + x + ")" 
    572620 
     621 
     622class StorageManagerDB(storage.StorageManager): 
     623    """StoreManager to save and retrieve Units using a DB.""" 
     624     
     625    identifier_length = 64 
     626    identifier_caseless = False 
     627    close_connection_method = 'close' 
     628    decompiler = SQLDecompiler 
     629    typeAdapter = FieldTypeAdapter() 
     630    toAdapter = AdapterToSQL() 
     631    fromAdapter = AdapterFromDB() 
     632     
     633    def __init__(self, name, arena, allOptions={}): 
     634        storage.StorageManager.__init__(self, name, arena, allOptions) 
     635         
     636        self.CreateIfMissing = allOptions.get(u'Create If Missing', '') 
     637        self.threaded = bool(allOptions.get(u'Threaded', 'True')) 
     638        self._connection = None 
     639        self.prefix = allOptions.get(u'Prefix', u"djv") 
     640        self.reserve_lock = threading.Lock() 
     641         
     642        ec = {} 
     643        for prop in allOptions.get(u'Expanded Columns', '').split(","): 
     644            if prop: 
     645                cls, type = [x.strip() for x in prop.split(":", 1)] 
     646                lastdot = cls.rfind(".") 
     647                clsname, key = cls[:lastdot], cls[lastdot + 1:] 
     648                ec[(clsname, key)] = type 
     649        self.expanded_columns = ec 
     650     
     651    def identifier(self, *atoms): 
     652        ident = ''.join(map(str, atoms)).replace('"', '""') 
     653        if self.identifier_caseless: 
     654            ident = ident.lower() 
     655        idlen = self.identifier_length 
     656        if idlen and len(ident) > idlen: 
     657            warnings.warn("Identifier is longer than %s characters." % idlen) 
     658            ident = ident[:idlen] 
     659        return '"' + ident + '"' 
     660     
     661    def tablename(self, cls): 
     662        if isinstance(cls, type): 
     663            name = cls.__name__ 
     664        elif isinstance(cls, dejavu.Unit): 
     665            name = cls.__class__.__name__ 
     666        elif isinstance(cls, basestring): 
     667            name = cls 
     668        else: 
     669            raise TypeError("Cannot form tablenames from %s" % cls) 
     670        return self.identifier(self.prefix, name) 
     671     
     672    def _get_conn(self): 
     673        # Override this with the connection call for your DB. Example follows: 
     674##        try: 
     675##            conn = libpq.PQconnectdb(self.connstring) 
     676##        except Exception, x: 
     677##            if self.CreateIfMissing: 
     678##                self.create_database() 
     679##                conn = libpq.PQconnectdb(self.connstring) 
     680##            else: 
     681##                raise 
     682##        return conn 
     683        raise NotImplementedError 
     684     
     685    def connection(self): 
     686        if self.threaded: 
     687            t = threading.currentThread() 
     688            if not hasattr(t, 'dejavu_storage_connection'): 
     689                t.dejavu_storage_connection = self._get_conn() 
     690            return t.dejavu_storage_connection 
     691        else: 
     692            if self._connection is None: 
     693                self._connection = self._get_conn() 
     694            return self._connection 
     695     
     696    def create_database(self): 
     697        self.execute('CREATE DATABASE %s;' % self.identifier(self.dbname)) 
     698     
     699    def drop_database(self): 
     700        self.execute("DROP DATABASE %s;" % self.identifier(self.dbname)) 
     701     
     702    def shutdown(self): 
     703        if self.threaded: 
     704            t = threading.currentThread() 
     705            conn = getattr(t, "dejavu_storage_connection", None) 
     706        else: 
     707            conn = self._connection 
     708         
     709        if self.threaded: 
     710            t = threading.currentThread() 
     711            conn = getattr(t, "dejavu_storage_connection", None) 
     712            t.dejavu_storage_connection = None 
     713        else: 
     714            conn = self._connection 
     715            self._connection = None 
     716         
     717        if conn is not None: 
     718            getattr(conn, self.close_connection_method)() 
     719            conn = None 
     720     
     721    def select(self, unitClass, expr, distinct_fields=None): 
     722        tablename = self.tablename(unitClass) 
     723        if distinct_fields: 
     724            distinct_fields = [self.identifier(x) for x in distinct_fields] 
     725            sql = (u'SELECT DISTINCT %s FROM %s' % 
     726                   (u', '.join(distinct_fields), tablename)) 
     727        else: 
     728            sql = u'SELECT * FROM %s' % tablename 
     729        w, i = self.where(unitClass, expr) 
     730        if len(w) > 0: 
     731            w = u" WHERE " + w 
     732        else: 
     733            w = u"" 
     734        sql += w + ";" 
     735        return sql, i 
     736     
     737    def where(self, cls, expr): 
     738        decom = self.decompiler(self.tablename(cls), expr, self.toAdapter) 
     739        return decom.code(), decom.imperfect 
     740     
     741    def execute(self, query, conn=None): 
     742        """execute(query, conn=None) -> result set.""" 
     743        if conn is None: 
     744            conn = self.connection() 
     745        try: 
     746            return conn.query(query) 
     747        except Exception, x: 
     748            x.args += (query,) 
     749            raise x 
     750     
     751    def fetch(self, query, conn=None): 
     752        """fetch(query, conn=None) -> rowdata, columns. 
     753         
     754        This base class uses SQLite3 syntax.""" 
     755        res = self.execute(query, conn) 
     756        return res.row_list, res.col_defs 
     757     
     758    def recall(self, cls, expr=None, pairs=None): 
     759        if expr is None: 
     760            expr = logic.Expression(lambda x: True) 
     761        sql, imperfect = self.select(cls, expr) 
     762        data, col_defs = self.fetch(sql) 
     763         
     764        columns = {} 
     765        for index, col in enumerate(col_defs): 
     766            # name, type_code, display_size, internal_size, precision, 
     767            # scale, null_ok 
     768            columns[col[0]] = (index, col[1]) 
     769         
     770        consume = self.fromAdapter.consume 
     771        for row in data: 
     772            unit = cls() 
     773             
     774            # Get the ID property first in case other fields depend upon it. 
     775            # See load_expanded, for example. 
     776            if self.identifier_caseless: 
     777                index, ftype = columns['id'] 
     778            else: 
     779                index, ftype = columns['ID'] 
     780            consume(unit, 'ID', row[index], ftype) 
     781             
     782            # Get remaining properties. 
     783            for key in cls.properties(): 
     784                if key != 'ID': 
     785                    if self.identifier_caseless: 
     786                        index, ftype = columns[key.lower()] 
     787                    else: 
     788                        index, ftype = columns[key] 
     789                    value = row[index] 
     790                     
     791                    subtype = self.expanded_columns.get((cls.__name__, key)) 
     792                    if subtype: 
     793                        self.load_expanded(unit, key, subtype) 
     794                    else: 
     795                        try: 
     796                            consume(unit, key, value, ftype) 
     797                        except Exception, x: 
     798                            x.args += (key, value, ftype) 
     799                            raise x 
     800             
     801            # If our SQL is imperfect, don't yield it to the 
     802            # caller unless it passes evaluate(). 
     803            if (not imperfect) or expr.evaluate(unit): 
     804                unit.cleanse() 
     805                yield unit 
     806     
     807    def reserve(self, unit): 
     808        """reserve(unit). -> Reserve a persistent slot for unit. 
     809         
     810        Notice in particular that we do not use the auto-number or 
     811        sequence generation capabilities within some databases, etc. 
     812        The ID should be supplied by UnitSequencers via reserve(). 
     813        """ 
     814        tablename = self.tablename(unit) 
     815        id = self.identifier("ID") 
     816        coerce = self.fromAdapter.coerce 
     817        expectedType = unit.__class__.property_type("ID") 
     818        self.reserve_lock.acquire() 
     819        try: 
     820            if unit.ID is None: 
     821                data, cols = self.fetch(u'SELECT %s FROM %s;' % (id, tablename)) 
     822                coltype = cols[0][1] 
     823                data = [coerce(row[0], coltype, expectedType) for row in data] 
     824                unit.ID = unit.sequencer.next(data) 
     825             
     826            self.execute('INSERT INTO %s (%s) VALUES (%s);' % 
     827                         (tablename, id, self.toAdapter.coerce(unit.ID))) 
     828        finally: 
     829            self.reserve_lock.release() 
     830     
     831    def save(self, unit, forceSave=False): 
     832        """save(unit, forceSave=False) -> Update storage from unit's data.""" 
     833        if unit.dirty() or forceSave: 
     834            cls = unit.__class__ 
     835             
     836            parms = [] 
     837            for key in cls.properties(): 
     838                if key != "ID": 
     839                    subtype = self.expanded_columns.get((cls.__name__, key)) 
     840                    if subtype: 
     841                        self.save_expanded(unit, key, subtype) 
     842                    else: 
     843                        val = self.toAdapter.coerce(getattr(unit, key)) 
     844                        parms.append('%s = %s' % (self.identifier(key), val)) 
     845             
     846            sql = ('UPDATE %s SET %s WHERE %s = %s;' % 
     847                   (self.tablename(unit), u", ".join(parms), 
     848                    self.identifier("ID"), 
     849                    self.toAdapter.coerce(unit.ID))) 
     850            self.execute(sql) 
     851            unit.cleanse() 
     852     
     853    def save_expanded(self, unit, key, subtype): 
     854        """save_expanded(unit, key, subtype). Save list in separate table.""" 
     855        unitcls = unit.__class__ 
     856        table = self.identifier(self.prefix, "_", unitcls.__name__, 
     857                                "_", unit.ID, "_", key) 
     858         
     859        # Just drop the old table and start with a new one. 
     860        try: 
     861            self.execute(u"DROP TABLE %s;" % table) 
     862        except: 
     863            pass 
     864         
     865        val = getattr(unit, key) 
     866        if val is None: 
     867            # Don't create a new table at all. This will signal 
     868            # recall() to set the attribute to None on load. 
     869            pass 
     870        else: 
     871            ftype = getattr(self.typeAdapter, "coerce_" + subtype)(unitcls, key) 
     872            self.execute(u"CREATE TABLE %s (EXPVAL %s);" % (table, ftype)) 
     873             
     874            for v in val: 
     875                self.execute(u"INSERT INTO %s (EXPVAL) VALUES ('%s');" 
     876                             % (table, self.toAdapter.coerce(v))) 
     877     
     878    def load_expanded(self, unit, key, subtype): 
     879        """load_expanded(unit, key, subtype). Load list from separate table.""" 
     880        table = self.identifier(self.prefix, "_", unit.__class__.__name__, 
     881                                "_", unit.ID, "_", key) 
     882        try: 
     883            data, col_defs = self.fetch(u"SELECT EXPVAL FROM %s" % table) 
     884        except: 
     885            # No table was made because no values were present. 
     886            # Feel free to narrow the exception in a subclass. 
     887            values = None 
     888        else: 
     889            coltype = col_defs[0][1] 
     890            coercer = getattr(self.fromAdapter, "coerce_" + subtype) 
     891            values = [coercer(row[0], coltype) for row in data] 
     892             
     893            expected_type = unit.__class__.property_type(key) 
     894            values = expected_type(values) 
     895         
     896        # Set the attribute directly to avoid __set__ overhead. 
     897        unit._properties[key] = values 
     898     
     899    def destroy(self, unit): 
     900        """Delete the unit.""" 
     901        self.execute(u'DELETE * FROM %s WHERE %s = %s;' % 
     902                     (self.tablename(unit), self.identifier("ID"), 
     903                      self.toAdapter.coerce(unit.ID))) 
     904     
     905    def create_storage(self, unitClass): 
     906        tablename = self.tablename(unitClass) 
     907         
     908        coerce = self.typeAdapter.coerce 
     909        fields = [] 
     910        for key in unitClass.properties(): 
     911            fields.append(u'%s %s' % (self.identifier(key), 
     912                                      coerce(unitClass, key))) 
     913        self.execute(u'CREATE TABLE %s (%s);' % (tablename, ", ".join(fields))) 
     914        for index in unitClass.indices(): 
     915            i = self.identifier(self.prefix, "i", unitClass.__name__, index) 
     916            self.execute(u'CREATE INDEX %s ON %s (%s);' % 
     917                         (i, tablename, self.identifier(index))) 
     918     
     919    def distinct(self, cls, fields, expr=None): 
     920        """Return distinct values for specified fields.""" 
     921        if expr is None: 
     922            expr = logic.Expression(lambda x: True) 
     923         
     924        sql, imperfect = self.select(cls, expr, fields) 
     925        if imperfect: 
     926            # ^%$#@! There's no way to handle imperfect queries without 
     927            # creating all involved Units, which defeats the purpose of 
     928            # distinct, which was a speed issue more than anything. Grr. 
     929            raise ValueError(u"The following query cannot be reliably " 
     930                             u"returned from this data source.", 
     931                             u"distinct()", cls, fields, expr) 
     932         
     933        data, columns = self.fetch(sql) 
     934        actualTypes = [x[1] for x in columns] 
     935        expectedTypes = [cls.property_type(x) for x in fields] 
     936         
     937        coerce = self.fromAdapter.coerce 
     938        # Must use inner tuples for hashability in Sandbox.distinct() 
     939        return [tuple([coerce(val, actualTypes[i], expectedTypes[i]) 
     940                       for i, val in enumerate(row)]) 
     941                 for row in data] 
     942 

  • trunk/storage/storeado.py

    r45 r46  
    66import pywintypes 
    77import pythoncom 
    8 import threading 
    98import datetime 
    109try: 
    …… 
    1211except ImportError: 
    1312    import pickle 
    14 from types import FunctionType 
    1513 
    1614try: 
    …… 
    2220from dejavu import storage, logic 
    2321from dejavu.storage import db 
    24 import recur 
     22from recur import sane_time 
    2523 
    2624adOpenForwardOnly = 0 
    …… 
    5654    mins = int(round(mins)) 
    5755    sec = int(round(sec)) 
    58     return recur.sane_time(0, hour, mins, sec) 
     56    return sane_time(0, hour, mins, sec) 
    5957 
    6058 
    …… 
    236234     
    237235    def column_name(self, name): 
    238         # This is valid SQL for PostgreSQL only and should be overridden. 
    239         return '[%s].[%s]' % (self.tablename, name) 
     236        return '%s.[%s]' % (self.tablename, name) 
    240237     
    241238    # --------------------------- Dispatchees --------------------------- # 
    …… 
    295292 
    296293 
    297  
    298 class StoreIteratorADO(object): 
    299     """Iterator for populating Units from storage.""" 
    300      
    301     def __init__(self, store, unitClass, expr): 
    302         self.store  = store 
    303         self.unitClass = unitClass 
    304         self.expr = expr 
    305         self.colIndices = {} 
    306         self.fieldTypes = [] 
    307          
    308         self.sql, self.imperfect = store.select(unitClass, expr) 
    309      
    310     def load_data(self): 
    311         anRS = self.store.recordset(self.sql, adOpenForwardOnly, 
    312                                     adLockReadOnly) 
    313          
    314         for col, x in enumerate(anRS.Fields): 
    315             self.colIndices[x.Name] = col 
    316             self.fieldTypes.append(x.Type) 
    317          
    318         self.data = [] 
    319         if not(anRS.BOF and anRS.EOF): 
    320             # We tried .MoveNext() and lots of Fields.Item() calls. 
    321             # Using GetRows() beats that time by about 2/3. 
    322             self.data = anRS.GetRows() 
    323         anRS.Close() 
    324      
    325     def units(self): 
    326         s = self.store 
    327         clsname = self.unitClass.__name__ 
    328         self.load_data() 
    329         if len(self.data) > 0: 
    330             for row in range(len(self.data[0])): 
    331                 unit = self.unitClass() 
    332                 coercer = AdapterFromADO(unit) 
    333                 for key in unit.__class__.properties(): 
    334                     if (clsname, key) in s.expanded_columns: 
    335                         # Grab the expanded data 
    336                         try: 
    337                             col = self.colIndices['ID'] 
    338                             ID = self.data[col][row] 
    339                             i = s.identifier(s.prefix, "_", clsname, 
    340                                              "_", ID, "_", key) 
    341                             rs = s.recordset(u"SELECT EXPVAL FROM %s" % i) 
    342                         except pywintypes.com_error, x: 
    343                             # This usually occurs because 1) the parent Unit 
    344                             # was reserved but no table yet made for these 
    345                             # expanded values, or 2) no table was made 
    346                             # because no values were present. This is OK. 
    347                             # TODO: trap this more specifically by examining 
    348                             # the errmsg. 
    349                             values = None 
    350                         else: 
    351                             if rs.BOF and rs.EOF: 
    352                                 values = [] 
    353                             else: 
    354                                 values = [pickle.loads(str(x)) 
    355                                           for x in rs.GetRows()[0]] 
    356                             rs.Close() 
    357                             expectedType = unit.__class__.property_type(key) 
    358                             values = expectedType(values) 
    359                         # Set the attribute directly to avoid __set__ overhead. 
    360                         unit._properties[key] = values 
    361                     else: 
    362                         try: 
    363                             col = self.colIndices[key] 
    364                         except KeyError, x: 
    365                             x.args += (key, self.unitClass.__name__) 
    366                             raise x 
    367                         coercer.consume(key, self.data[col][row], 
    368                                         self.fieldTypes[col]) 
    369                 # If our SQL is imperfect, don't yield it to the 
    370                 # caller unless it passes evaluate(). 
    371                 if (not self.imperfect) or self.expr.evaluate(unit): 
    372                     yield unit 
    373  
    374  
    375 class StoreMultiIteratorADO(StoreIteratorADO): 
     294class StoreMultiIteratorADO(object): 
    376295    """Iterator for populating Units (from multiple classes) from storage.""" 
    377296     
    …… 
    388307    def populate_unit(self, unit, row): 
    389308        """Populate a Unit from a database row.""" 
    390         coercer = AdapterFromADO(unit) 
     309        consume = AdapterFromADO().consume 
    391310        cls = unit.__class__ 
    392311        for key in cls.properties(): 
    …… 
    397316                raise x 
    398317            else: 
    399                 coercer.consume(key, self.data[col][row], self.fieldTypes[col]) 
     318                consume(unit, key, self.data[col][row], self.fieldTypes[col]) 
    400319     
    401320    def load_data(self): 
    402         anRS = self.store.recordset(self.sql, adOpenForwardOnly, 
    403                                     adLockReadOnly) 
     321        anRS = self.store.fetch(self.sql) 
    404322         
    405323        for col, x in enumerate(anRS.Fields): 
    …… 
    432350 
    433351 
    434 class StorageManagerADO(storage.StorageManager): 
     352class StorageManagerADO(db.StorageManagerDB): 
    435353    """StoreManager to save and retrieve Units via ADO 2.7. 
    436354     
    …… 
    438356    """ 
    439357     
     358    close_connection_method = 'Close' 
    440359    decompiler = ADOSQLDecompiler 
    441     createAdapter = db.FieldTypeAdapter() 
    442     threaded = False 
     360    toAdapter = AdapterToADOSQL() 
     361    fromAdapter = AdapterFromADO() 
    443362     
    444363    def __init__(self, name, arena, allOptions={}): 
    445364        pythoncom.CoInitialize() 
    446365         
    447         storage.StorageManager.__init__(self, name, arena, allOptions) 
     366        db.StorageManagerDB.__init__(self, name, arena, allOptions) 
    448367         
    449368        self.connstring = allOptions[u'Connect'] 
    450         self.CreateIfMissing = allOptions.get(u'Create If Missing', '') 
    451          
    452369        atoms = dict([pair.lower().split("=", 1) 
    453370                      for pair in self.connstring.split(";") 
    454371                      if pair]) 
    455         self.DBName = atoms.get(u'data source', None) 
    456          
    457         self.threaded = bool(allOptions.get(u'Threaded', '')) 
    458         self._connection = None 
    459          
    460         self.prefix = allOptions.get(u'Prefix', u"djv") 
     372        self.dbname = atoms.get(u'data source') or atoms.get(u'data source name') 
    461373        self.cursorType = int(allOptions.get(u'CursorType', adOpenDynamic)) 
    462374        self.lockType = int(allOptions.get(u'LockType', adLockOptimistic)) 
    463          
    464         ec = [] 
    465         for prop in allOptions.get(u'Expanded Columns', '').split(","): 
    466             if prop: 
    467                 lastdot = prop.rfind(".") 
    468                 clsname, key = prop[:lastdot], prop[lastdot + 1:] 
    469                 ec.append((clsname, key)) 
    470         self.expanded_columns = ec 
    471          
    472         self.reserve_lock = threading.Lock() 
    473375     
    474376    def identifier(self, *atoms): 
    475         ident = "[" + ''.join(map(str, atoms)) + "]" 
    476         return ident 
    477      
    478     def shutdown(self): 
    479         if self._connection is not None: 
    480             self._connection.Close() 
     377        ident = ''.join(map(str, atoms)) 
     378        return '[' + ident + ']' 
    481379     
    482380    def _get_conn(self): 
    …… 
    492390        return conn 
    493391     
    494     def connection(self): 
    495         if self.threaded: 
    496             t = threading.currentThread() 
    497             if not hasattr(t, 'dejavu_storage_connection'): 
    498                 t.dejavu_storage_connection = self._get_conn() 
    499             return t.dejavu_storage_connection 
    500         else: 
    501             if self._connection is None: 
    502                 self._connection = self._get_conn() 
    503             return self._connection 
    504      
    505     def recordset(self, aQuery, cursorType=None, lockType=None): 
    506         anRS = win32com.client.Dispatch(r'ADODB.Recordset') 
    507         # Uncomment the following line to obtain .Recordcount 
    508         # anRS.Cursorlocation = adUseClient 
    509         if cursorType is None: 
    510             cursorType = self.cursorType 
    511         if lockType is None: 
    512             lockType = self.lockType 
    513          
    514         try: 
    515             anRS.Open(aQuery, self.connection(), cursorType, lockType) 
    516         except pywintypes.com_error, x: 
    517             try: 
    518                 anRS.Close() 
    519             except: 
    520                 pass 
    521             x.args += (aQuery, ) 
    522             raise x 
    523         return anRS 
    524      
    525392    def _join(self, path=[]): 
    526         if not path: return u'' 
     393        if not path: 
     394            return u'' 
    527395        firstcls = path.pop(0) 
    528         if not path: return firstcls.__name__ 
     396        if not path: 
     397            return firstcls.__name__ 
    529398         
    530399        spath = self.arena.associations.shortest_path(firstcls, path[0]) 
    531400        spath.pop(0) 
    532401        cls = spath[0] 
     402         
     403        p = self.prefix 
     404        i = self.identifier 
     405        left = i(p, firstcls.__name__) 
     406        right = i(p, cls.__name__) 
     407        if len(spath) == 1: 
     408            child = right 
     409        else: 
     410            child = u"(%s)" % self._join(spath) 
    533411        leftkey, rightkey = firstcls._associations[cls] 
    534         params = {u'prefix': self.prefix, 
    535                   u'left': firstcls.__name__, 
    536                   u'right': cls.__name__, 
    537                   u'leftkey': leftkey, 
    538                   u'rightkey': rightkey, 
    539                   } 
    540         if len(spath) == 1: 
    541             params[u'child'] = self.identifier(params['prefix'], params['right']) 
    542         else: 
    543             params[u'child'] = u"(%s)" % self._join(spath) 
    544          
    545         return (u"[%(prefix)s%(left)s] LEFT JOIN %(child)s" 
    546                 u" ON [%(prefix)s%(left)s].[%(leftkey)s] = " 
    547                 u"[%(prefix)s%(right)s].[%(rightkey)s]" % params) 
     412         
     413        return (u"%s LEFT JOIN %s ON %s.%s = %s.%s" % 
     414                (left, child, left, leftkey, right, rightkey)) 
    548415     
    549416    def multiselect(self, firstcls, firstexpr, pairs): 
    …… 
    553420        if len(pairs) != 1: 
    554421            raise ValueError("Multiselect does not yet work on multiple pairs.") 
     422         
    555423        for cls, expr in pairs: 
    556424            if expr is None: 
    557425                expr = logic.Expression(lambda x: True) 
     426             
    558427            j = self._join([firstcls, cls]) 
    559428             
    …… 
    574443            return statement, imp, cols 
    575444     
    576     def select(self, unitClass, expr, distinct_fields=None): 
    577         tablename = self.identifier(self.prefix, unitClass.__name__) 
    578         if distinct_fields: 
    579             distinct_fields = [self.identifier(x) for x in distinct_fields] 
    580             sql = (u"SELECT DISTINCT %s FROM %s" % 
    581                    (u', '.join(distinct_fields), tablename)) 
    582         else: 
    583             sql = u"SELECT * FROM %s" % tablename 
    584         w, i = self.where(unitClass, expr) 
    585         if len(w) > 0: 
    586             w = u" WHERE " + w 
    587         else: 
    588             w = u"" 
    589         sql += w 
    590         return sql, i 
    591      
    592     def where(self, cls, expr): 
    593         decom = self.decompiler(self.prefix + cls.__name__, expr, AdapterToADOSQL) 
    594         return decom.code(), decom.imperfect 
    595      
    596     def execute(self, aQuery, conn=None): 
     445    def execute(self, query, conn=None): 
    597446        if conn is None: 
    598447            conn = self.connection() 
    599448        try: 
    600             conn.Execute(aQuery) 
     449            conn.Execute(query) 
    601450        except pywintypes.com_error, x: 
    602             x.args += (aQuery, ) 
     451            x.args += (query, ) 
    603452            raise x 
    604453     
    605     def recall(self, cls, expr=None, pairs=None): 
    606         if expr is None: 
    607             expr = logic.Expression(lambda x: True) 
    608          
    609         if pairs is not None: 
    610             return StoreMultiIteratorADO(self, cls, expr, pairs).units() 
    611         else: 
    612             return StoreIteratorADO(self, cls, expr).units() 
    613      
    614     def reserve(self, unit): 
    615         """reserve(unit). -> Reserve a persistent slot for unit.""" 
    616         self.reserve_lock.acquire() 
     454    def fetch(self, query, conn=None): 
     455        """fetch(query, conn=None) -> rowdata, columns.""" 
     456        if conn is None: 
     457            conn = self.connection() 
     458         
     459        res = win32com.client.Dispatch(r'ADODB.Recordset') 
     460        # Uncomment the following to get .Recordcount 
     461        # res.Cursorlocation = adUseClient 
    617462        try: 
    618             if unit.ID is None: 
    619                 data = [] 
    620                 clsname = unit.__class__.__name__ 
    621                 anRS = self.recordset(u"SELECT ID FROM %s;" % 
    622                                       self.identifier(self.prefix, clsname)) 
    623                 if not (anRS.BOF and anRS.EOF): 
    624                     data = anRS.GetRows()[0] 
    625                 unit.ID = unit.sequencer.next(data) 
    626                  
    627                 anRS.AddNew() 
    628                 anRS.Fields(u'ID').Value = unit.ID 
    629                 anRS.Update() 
    630                 anRS.Close() 
    631         finally: 
    632             self.reserve_lock.release() 
    633      
    634     def save(self, unit, forceSave=False): 
    635         """save(unit, forceSave=False). -> Update storage from unit's data. 
    636          
    637         Notice in particular that we do not use the auto-number or 
    638         sequence generation capabilities within some databases, etc. 
    639         The ID should be supplied by UnitSequencers via reserve(). 
    640         """ 
    641         if unit.dirty() or forceSave: 
    642             cls = unit.__class__ 
    643             clsname = cls.__name__ 
    644             # Use a cursor always--makes mixed-quotes, newline, etc easier. 
    645             anRS = self.recordset("SELECT * FROM %s WHERE ID = %s" % 
    646                                   (self.identifier(self.prefix, clsname), 
    647                                    AdapterToADOSQL().coerce(unit.ID))) 
    648             if anRS.EOF and anRS.BOF: 
    649                 anRS.AddNew() 
    650                 anRS.Fields(u'ID').Value = unit.ID 
    651             fmt = AdapterToADOFields() 
    652             for key in cls.properties(): 
    653                 if (clsname, key) in self.expanded_columns: 
    654                     # Special-case this field into its own table. 
    655                     self.save_expanded(unit, key) 
    656                 else: 
    657                     eachType = cls.property_type(key) 
    658                     newValue = fmt.coerce(getattr(unit, key), eachType) 
    659                     try: 
    660                         anRS.Fields(key).Value = newValue 
    661                     except pywintypes.com_error, x: 
    662                         try: 
    663                             anRS.Close() 
    664                         except: 
    665                             pass 
    666                         x.args += (clsname, key, eachType, newValue) 
    667                         raise x 
    668             anRS.Update() 
    669             anRS.Close() 
    670             unit.cleanse() 
    671      
    672     def save_expanded(self, unit, key): 
    673         """Save a field using a table specifically for that purpose.""" 
    674         unitcls = unit.__class__ 
    675         table = self.identifier(self.prefix, "_", unitcls.__name__, "_", 
    676                                 unit.ID, "_", key) 
    677          
    678         conn = self.connection() 
    679          
    680         # Just drop the old table and start with a new one. 
    681         try: 
    682             self.execute((u"DROP TABLE %s;" % table), conn) 
     463            res.Open(query, conn, adOpenForwardOnly, adLockReadOnly) 
    683464        except pywintypes.com_error, x: 
    684             pass 
    685          
    686         val = getattr(unit, key) 
    687         if val is None: 
    688             # Don't create a new table at all. This will signal 
    689             # our iterator to set the attribute to None on load. 
    690             pass 
    691         else: 
    692             # Ugly, ugly hack to get NTEXT or MEMO as appropriate. The point 
    693             # is, we want a large text field so we can pickle each item. 
    694             ftype = self.createAdapter.coerce_list(unitcls, key) 
    695             self.execute(u"CREATE TABLE %s (EXPVAL %s);" % (table, ftype), conn) 
    696              
    697             ins = u"INSERT INTO " + table + " (EXPVAL) VALUES ('%s');" 
    698             for v in val: 
    699                 # Create a row for the unit. 
    700                 # Use an INSERT command (not a cursor) for better performance. 
    701                 v = pickle.dumps(v).replace("'", "''") 
    702                 self.execute(ins % v, conn) 
    703      
    704     def destroy(self, unit): 
    705         """Delete the unit.""" 
    706         # Use a DELETE command instead of a cursor for better performance. 
    707         deleteStatement = (u"DELETE * FROM %s WHERE ID = %s" % 
    708                            (self.identifier(self.prefix, unit.__class__.__name__), 
    709                             AdapterToADOSQL().coerce(unit.ID))) 
    710         self.execute(deleteStatement) 
    711      
    712     def create_storage(self, unitClass): 
    713         coerce = self.createAdapter.coerce 
    714         fields = [] 
    715         for key in unitClass.properties(): 
    716             if (unitClass.__name__, key) not in self.expanded_columns: 
    717                 fields.append(u"%s %s" % (self.identifier(key), 
    718                                           coerce(unitClass, key))) 
    719         self.execute(u"CREATE TABLE %s (%s)" % 
    720                      (self.identifier(self.prefix, unitClass.__name__), 
    721                       ", ".join(fields))) 
    722          
    723         for index in unitClass.indices(): 
    724             self.execute(u"CREATE INDEX %s ON %s (%s ASC)" 
    725                          % (self.identifier(self.prefix, "i", 
    726                                             unitClass.__name__, index), 
    727                             self.identifier(self.prefix, unitClass.__name__), 
    728                             index)) 
    729      
    730     def distinct(self, cls, fields, expr=None): 
    731         """Return distinct values for specified fields.""" 
    732         if expr is None: 
    733             expr = logic.Expression(lambda x: True) 
    734          
    735         # ^%$#@! There's no way to handle imperfect queries without 
    736         # creating all involved Units, which defeats the purpose of 
    737         # distinct, which was a speed issue more than anything. Grr. 
    738         sql, imperfect = self.select(cls, expr, fields) 
    739         # Ignore for now. 
    740 ##        if imperfect: 
    741 ##            raise ValueError(u"The following query cannot be reliably " 
    742 ##                             u"returned from an ADO data source.", 
    743 ##                             u"distinct()", cls, fields, expr) 
    744          
    745         anRS = self.recordset(sql, adOpenForwardOnly, adLockReadOnly) 
    746          
    747         fieldTypes = [x.Type for x in anRS.Fields] 
     465            try: 
     466                res.Close() 
     467            except: 
     468                pass 
     469            x.args += (query, ) 
     470            raise x 
     471         
     472        columns = [(x.Name, x.Type) for x in res.Fields] 
     473         
    748474        data = [] 
    749         if not (anRS.BOF and anRS.EOF): 
     475        if not(res.BOF and res.EOF): 
    750476            # We tried .MoveNext() and lots of Fields.Item() calls. 
    751477            # Using GetRows() beats that time by about 2/3. 
    752             data = anRS.GetRows() 
    753         anRS.Close() 
    754          
    755         if data: 
    756             coerced_data = [] 
    757             coerce = AdapterFromADO().coerce 
    758             for col, field in enumerate(fields): 
    759                 expectedType = cls.property_type(field) 
    760                 actualType = fieldTypes[col] 
    761                 coerced_row = [coerce(val, actualType, expectedType) 
    762                                for val in data[col]] 
    763                 coerced_data.append(coerced_row) 
    764             data = zip(*coerced_data) 
    765         return data 
     478            data = res.GetRows() 
     479            # Convert cols x rows -> rows x cols 
     480            data = zip(*data) 
     481        res.Close() 
     482         
     483        return data, columns 
    766484 
    767485 
    …… 
    775493class FieldTypeAdapter_SQLServer(db.FieldTypeAdapter): 
    776494     
     495    numeric_max_precision = 38 
     496     
    777497    def coerce_str(self, cls, key): 
    778         # The bytes hint shall not reflect the usual 4-byte base for varchar. 
     498        # The bytes hint does not reflect the usual 4-byte base for varchar. 
    779499        prop = getattr(cls, key) 
    780500        bytes = int(prop.hints.get(u'bytes', '0')) 
    781         if bytes and bytes < 8000: 
    782             return u"VARCHAR(%s)" % bytes 
    783         else: 
     501        if bytes and bytes <= 8000: 
    784502            # 8000 *bytes* is the absolute upper limit, based on T_SQL docs 
    785503            # for varchar. If there are further fields defined for the class, 
    …… 
    789507            # now, we just trust that units generally use a size of 0 to 
    790508            # bump up to NTEXT (1 gig characters). 
     509            return u"VARCHAR(%s)" % bytes 
     510        else: 
    791511            return u"NTEXT" 
    792512 
    793513 
    794514class StorageManagerADO_SQLServer(StorageManagerADO): 
    795     createAdapter = FieldTypeAdapter_SQLServer() 
     515     
     516    typeAdapter = FieldTypeAdapter_SQLServer() 
    796517     
    797518    def create_database(self): 
    …… 
    805526                        atoms.get('PASSWORD') or atoms.get('PWD')) 
    806527                     ) 
    807         adoconn.Execute("CREATE DATABASE %s" % self.DBName) 
     528        adoconn.Execute("CREATE DATABASE %s" % self.dbname) 
    808529 
    809530 
    …… 
    830551class FieldTypeAdapter_MSAccess(db.FieldTypeAdapter): 
    831552     
     553    numeric_max_precision = 28 
     554     
    832555    def coerce_str(self, cls, key): 
    833556        # The bytes hint shall not reflect the usual 4-byte base for varchar. 
    834557        prop = getattr(cls, key) 
    835558        bytes = int(prop.hints.get(u'bytes', '0')) 
    836         if bytes and bytes < 8000: 
     559        if bytes and bytes <= 255: 
     560            # 255 chars is the upper limit for TEXT / VARCHAR in MS Access. 
    837561            return u"VARCHAR(%s)" % bytes 
    838562        else: 
    839             # 255 chars is the upper limit for TEXT / VARCHAR in MS Access. 
    840563            # MEMO is 1 gigabyte when set programatically (only 64K when set 
    841564            # in Access UI). But then, 1 GB is the limit for the whole DB. 
    …… 
    846569     
    847570    decompiler = ADOSQLDecompiler_MSAccess 
    848     createAdapter = FieldTypeAdapter_MSAccess() 
     571    typeAdapter = FieldTypeAdapter_MSAccess() 
    849572     
    850573    def create_database(self): 
    …… 
    854577     
    855578    def drop_database(self): 
    856         atoms = dict([pair.upper().split("=", 1) 
    857                       for pair in self.connstring.split(";") 
    858                       if pair]) 
    859         dbname = atoms.get("DATA SOURCE") or atoms.get("DATA SOURCE NAME") 
    860579        import os 
    861580        # This should accept relative or absolute paths 
    862         os.remove(dbname) 
     581        os.remove(self.dbname) 
    863582 
    864583 

  • trunk/storage/storemysql.py

    r45 r46  
    1111# Use _mysql directly to avoid all of the DB-API overhead. 
    1212import _mysql 
    13  
    14 import warnings 
     13import datetime 
    1514import threading 
    16 import datetime 
    17  
    18 import dejavu 
    1915from dejavu import storage, logic 
    2016from dejavu.storage import db 
    21  
    22  
    23 AdapterToMySQL = db.AdapterToSQL() 
    24 AdapterFromMySQL = db.AdapterFromDB 
    2517 
    2618 
    …… 
    2921    def column_name(self, name): 
    3022        # MySQL forces lowercase column names. 
    31         return '`%s`.`%s`' % (self.tablename, name.lower()) 
     23        return '%s.`%s`' % (self.tablename, name.lower()) 
    3224     
    3325    # --------------------------- Dispatchees --------------------------- # 
    …… 
    6456 
    6557 
    66 class StoreIteratorMySQL(object): 
    67     """Iterator for populating Units from storage.""" 
    68      
    69     def __init__(self, store, unitClass, expr): 
    70         self.store  = store 
    71         self.unitClass = unitClass 
    72         self.expr = expr 
    73         self.sql, self.imperfect = store.select(unitClass, expr) 
    74      
    75     def units(self): 
    76         s = self.store 
    77          
    78         res = s.execute(self.sql) 
    79         if res.num_rows(): 
    80             columns = {} 
    81             for index, col in enumerate(res.describe()): 
    82                 # name, type_code, display_size, internal_size, precision, scale, null_ok 
    83                 columns[col[0].lower()] = (index, col[1]) 
    84              
    85             data = res.fetch_row(0, 0) 
    86              
    87             for row in data: 
    88                 unit = self.unitClass() 
    89                 coercer = AdapterFromMySQL(unit) 
    90                 for key in unit.__class__.properties(): 
    91                     index, ftype = columns[key.lower()] 
    92                     value = row[index] 
    93                     try: 
    94                         coercer.consume(key, value, ftype) 
    95                     except Exception, x: 
    96                         x.args += (key, value, ftype) 
    97                         raise x 
    98                 # If our SQL is imperfect, don't yield it to the 
    99                 # caller unless it passes evaluate(). 
    100                 if (not self.imperfect) or self.expr.evaluate(unit): 
    101                     yield unit 
    102  
    103  
    10458class FieldTypeAdapterMySQL(db.FieldTypeAdapter): 
    10559    """Return the SQL typename of a DB column.""" 
     60     
     61    # This was determined through experimentation. Don't change it. 
     62    numeric_max_precision = 253 
    10663     
    10764    def coerce_str(self, cls, key): 
    …… 
    12380 
    12481 
    125 class StorageManagerMySQL(storage.StorageManager): 
    126     """StoreManager to save and retrieve Units via _mysql .""" 
     82class StorageManagerMySQL(db.StorageManagerDB): 
     83    """StoreManager to save and retrieve Units via _mysql.""" 
    12784     
    128     createAdapter = FieldTypeAdapterMySQL() 
    129     threaded = False 
     85    identifier_length = 64 
     86    identifier_caseless = True 
     87    typeAdapter = FieldTypeAdapterMySQL() 
    13088     
    13189    def __init__(self, name, arena, allOptions={}): 
    132         storage.StorageManager.__init__(self, name, arena, allOptions) 
     90        db.StorageManagerDB.__init__(self, name, arena, allOptions) 
    13391         
    13492        connargs = ["host", "user", "passwd", "db", "port", "unix_socket", 
    …… 
    13997        self.connargs = dict([(k, v) for k, v in allOptions.iteritems() 
    14098                              if k in connargs]) 
    141          
    142         self.CreateIfMissing = allOptions.get(u'Create If Missing', '') 
    143         self.threaded = bool(allOptions.get(u'Threaded', '1')) 
    144         self._connection = None 
    145          
    146         self.prefix = allOptions.get(u'Prefix', u"djv") 
    147         self.reserve_lock = threading.Lock() 
     99        self.dbname = self.connargs['db'] 
    148100         
    149101        self.decompiler = MySQLDecompiler 
    150102        # Try to get the version string from MySQL, to see if we need 
    151103        # a different decompiler. 
    152         try: 
    153             res = self.execute("SELECT VERSION();") 
    154             if res.num_rows(): 
    155                 version = storage.Version(res.fetch_row(1, 0)[0][0]) 
    156                 if version > storage.Version("4.1.1"): 
    157                     self.decompiler = MySQLDecompiler411 
    158         except: 
    159             pass 
     104        data, columns = self.fetch("SELECT VERSION();") 
     105        if data: 
     106            version = storage.Version(data[0][0]) 
     107            if version > storage.Version("4.1.1"): 
     108                self.decompiler = MySQLDecompiler411 
    160109     
    161110    def identifier(self, *atoms): 
    162         # MySQL uses case-sensitive database and table names on Unix, but  
     111        # MySQL uses case-sensitive database and table names on Unix, but 
    163112        # not on Windows. Use all-lowercase identifiers to work around the 
    164113        # problem. "Column names, index names, and column aliases are not 
    165114        # case sensitive on any platform." 
    166115        # If deployers set lower_case_table_names to 1, it would help. 
    167         ident = "`" + ''.join(map(str, atoms)).replace("`", "``").lower() + "`" 
    168         if len(ident) > 64: 
    169             warnings.warn("Identifier is longer than 64 characters. MySQL " 
    170                           "identifiers are limited to 64 bytes.") 
    171         return ident 
    172      
    173     def shutdown(self): 
    174         if self.threaded: 
    175             t = threading.currentThread() 
    176             conn = getattr(t, "dejavu_storage_connection", None) 
    177             if conn is not None: 
    178                 conn.close() 
    179         else: 
    180             if self._connection is not None: 
    181                 self._connection.close() 
     116        ident = ''.join(map(str, atoms)).replace('`', '``').lower() 
     117        idlen = self.identifier_length 
     118        if idlen and len(ident) > idlen: 
     119            warnings.warn("Identifier is longer than %s characters." % idlen) 
     120            ident = ident[:idlen] 
     121        return '`' + ident + '`' 
    182122     
    183123    def _get_conn(self): 
    …… 
    192132        return conn 
    193133     
    194     def connection(self): 
    195         if self.threaded: 
    196             t = threading.currentThread() 
    197             if not hasattr(t, 'dejavu_storage_connection'): 
    198                 t.dejavu_storage_connection = self._get_conn() 
    199             return t.dejavu_storage_connection 
    200         else: 
    201             if self._connection is None: 
    202                 self._connection = self._get_conn() 
    203             return self._connection 
     134    def _template_conn(self): 
     135        tmplconn = self.connargs.copy() 
     136        tmplconn['db'] = '' 
     137        return _mysql.connect(**tmplconn) 
    204138     
    205139    def create_database(self): 
    206         tmplconn = self.connargs.copy() 
    207         tmplconn['db'] = '' 
    208         conn = _mysql.connect(**tmplconn) 
    209         self.execute('CREATE DATABASE %s;' % 
    210                      self.identifier(self.connargs['db']), conn) 
     140        sql = 'CREATE DATABASE %s;' % self.identifier(self.dbname) 
     141        conn = self._template_conn() 
     142        self.execute(sql, conn) 
    211143        conn.close() 
    212144     
    213145    def drop_database(self): 
    214         self.execute("DROP DATABASE %s;" % 
    215                      self.identifier(self.connargs['db'])) 
     146        sql = 'DROP DATABASE %s;' % self.identifier(self.dbname) 
     147        conn = self._template_conn() 
     148        self.execute(sql, conn) 
     149        conn.close() 
    216150     
    217     def select(self, unitClass, expr, distinct_fields=None): 
    218         tablename = self.identifier(self.prefix, unitClass.__name__) 
    219         if distinct_fields: 
    220             distinct_fields = [self.identifier(x) for x in distinct_fields] 
    221             sql = (u'SELECT DISTINCT %s FROM %s' % 
    222                    (u', '.join(distinct_fields), tablename)) 
     151    def shutdown(self): 
     152        if self.threaded: 
     153            t = threading.currentThread() 
     154            conn = getattr(t, "dejavu_storage_connection", None) 
     155            if conn is not None: 
     156                conn.close() 
    223157        else: 
    224             sql = u'SELECT * FROM %s' % tablename 
    225         w, i = self.where(unitClass, expr) 
    226         if len(w) > 0: 
    227             w = u" WHERE " + w 
    228         else: 
    229             w = u"" 
    230         sql += w 
    231         return sql, i 
     158            if self._connection is not None: 
     159                self._connection.close() 
    232160     
    233     def where(self, cls, expr): 
    234         tablename = self.prefix + cls.__name__ 
    235         decom = self.decompiler(tablename.lower(), expr) 
    236         return decom.code(), decom.imperfect 
    237      
    238     def execute(self, query, conn=None): 
     161    def fetch(self, query, conn=None): 
     162        """fetch(query, conn=None) -> rowdata, columns. 
     163         
     164        rowdata: a nested list (or tuples), column values within rows. 
     165        columns: a series of 2-tuples (or more). The first tuple value 
     166            will be the column name, the second value will be the column 
     167            type. 
     168        """ 
    239169        if conn is None: 
    240170            conn = self.connection() 
    241         try: 
    242             conn.query(query) 
    243             # store_result uses a client-side cursor 
    244             return conn.store_result() 
    245         except Exception, x: 
    246             x.args += (query,) 
    247             raise x 
    248      
    249     def recall(self, cls, expr=None, pairs=None): 
    250         if expr is None: 
    251             expr = logic.Expression(lambda x: True) 
    252         return StoreIteratorMySQL(self, cls, expr).units() 
    253      
    254     def reserve(self, unit): 
    255         """reserve(unit). -> Reserve a persistent slot for unit. 
    256          
    257         Notice in particular that we do not use the auto-number or 
    258         sequence generation capabilities within some databases, etc. 
    259         The ID should be supplied by UnitSequencers via reserve(). 
    260         """ 
    261         clsname = unit.__class__.__name__ 
    262         tblname = self.identifier(self.prefix, clsname) 
    263         id = self.identifier("ID") 
    264         self.reserve_lock.acquire() 
    265         try: 
    266             coerce = AdapterFromMySQL(unit).coerce 
    267             expectedType = unit.__class__.property_type("ID") 
    268             if unit.ID is None: 
    269                 data = [] 
    270                 res = self.execute(u'SELECT %s FROM %s;' % (id, tblname)) 
    271                 if res.num_rows(): 
    272                     data = [coerce(x[0], type(x[0]), expectedType) 
    273                             for x in res.fetch_row(0, 0)] 
    274                 unit.ID = unit.sequencer.next(data) 
    275              
    276             self.execute('INSERT INTO %s (%s) VALUES (%s);' % 
    277                          (tblname, id, AdapterToMySQL.coerce(unit.ID))) 
    278         finally: 
    279             self.reserve_lock.release() 
    280      
    281     def save(self, unit, forceSave=False): 
    282         """save(unit, forceSave=False) -> Update storage from unit's data.""" 
    283         if unit.dirty() or forceSave: 
    284             cls = unit.__class__ 
    285             tablename = self.identifier(self.prefix, cls.__name__) 
    286              
    287             parms = [] 
    288             for key in cls.properties(): 
    289                 if key != "ID": 
    290                     val = AdapterToMySQL.coerce(getattr(unit, key)) 
    291                     parms.append('%s = %s' % (self.identifier(key), val)) 
    292             sql = ('UPDATE %s SET %s WHERE %s = %s' % 
    293                    (tablename, u", ".join(parms), self.identifier("ID"), 
    294                     AdapterToMySQL.coerce(unit.ID, cls.property_type("ID")))) 
    295             self.execute(sql) 
    296             unit.cleanse() 
    297      
    298     def destroy(self, unit): 
    299         """Delete the unit.""" 
    300         # Use a DELETE command instead of a cursor for better performance. 
    301         deleteStatement = (u'DELETE * FROM %s WHERE %s = %s' % 
    302                            (self.identifier(self.prefix, unit.__class__.__name__), 
    303                             self.identifier("ID"), 
    304                             AdapterToMySQL.coerce(unit.ID))) 
    305         self.execute(deleteStatement) 
    306      
    307     def create_storage(self, unitClass): 
    308         clsname = unitClass.__name__ 
    309         tblname = self.identifier(self.prefix, clsname) 
    310          
    311         coerce = self.createAdapter.coerce 
    312         fields = [] 
    313         for key in unitClass.properties(): 
    314             fields.append(u'%s %s' % (self.identifier(key), coerce(unitClass, key))) 
    315          
    316         try: 
    317             self.execute(u'CREATE TABLE %s (%s)' % (tblname, ", ".join(fields))) 
    318         except _mysql.OperationalError, x: 
    319             if not x.args[1].endswith(" already exists"): 
    320                 raise 
    321         else: 
    322             for index in unitClass.indices(): 
    323                 indexname = self.identifier(self.prefix, "i", clsname, index) 
    324                 self.execute(u'CREATE INDEX %s ON %s (%s)' 
    325                              % (indexname, tblname, self.identifier(index))) 
    326      
    327     def distinct(self, cls, fields, expr=None): 
    328         """Return distinct values for specified fields.""" 
    329         if expr is None: 
    330             expr = logic.Expression(lambda x: True) 
    331          
    332         # ^%$#@! There's no way to handle imperfect queries without 
    333         # creating all involved Units, which defeats the purpose of 
    334         # distinct, which was a speed issue more than anything. Grr. 
    335         sql, imperfect = self.select(cls, expr, fields) 
    336         # Ignore for now. 
    337 ##        if imperfect: 
    338 ##            raise ValueError(u"The following query cannot be reliably " 
    339 ##                             u"returned from a Postgres data source.", 
    340 ##                             u"distinct()", cls, fields, expr) 
    341          
    342         res = self.execute(sql) 
    343         if res.num_rows(): 
    344             data = [] 
    345             coerce = AdapterFromMySQL().coerce 
    346             for row in data: 
    347                 coerced_row = [] 
    348                 for i in xrange(len(fields)): 
    349                     expectedType = cls.property_type(field[i]) 
    350                     actualType = res.ftype(i) 
    351                     val = coerce(res.getvalue(row, i), actualType, expectedType) 
    352                     coerced_row.append(val) 
    353                 data.append(coerced_row) 
    354             return zip(*data) 
    355         else: 
    356             return [] 
     171        self.execute(query, conn) 
     172        # store_result uses a client-side cursor 
     173        res = conn.store_result() 
     174        return res.fetch_row(0, 0), res.describe() 
    357175 

  • trunk/storage/storeodbc.py

    r45 r46  
    1 """This module is seriously broken; it hasn't been updated to match 
    2 framework redesign. But we didn't want to destroy what we learned so far. 
    3 """ 
     1"""ODBC Storage Manager for Dejavu.""" 
    42 
    53import dbi, odbc 
    6  
     4import threading 
    75import warnings 
    8 import threading 
    96import datetime 
    10  
    11 try: 
    12     import fixedpoint 
    13 except ImportError: 
    14     pass 
    15  
    16 import dejavu 
    17 from dejavu import storage, logic 
     7from dejavu import logic 
    188from dejavu.storage import db 
    199 
    2010 
    21 AdapterToPgSQL = db.AdapterToSQL() 
    22 AdapterFromPg = db.AdapterFromDB 
     11class FieldTypeAdapterODBC(db.FieldTypeAdapter): 
     12     
     13    # dbiDate objects have limited range. Use strings instead. 
     14    def coerce_datetime_datetime(self, cls, key): return u"CHAR(19)" 
     15    def coerce_datetime_date(self, cls, key): return u"CHAR(10)" 
     16    def coerce_datetime_time(self, cls, key): return u"CHAR(8)" 
     17     
     18    def coerce_decimal_Decimal(self, cls, key): 
     19        warnings.warn("The precision of %s.%s cannot be determined for " 
     20                      "ODBC stores. Values may be stored incorrectly." 
     21                      % (cls.__name__, key)) 
     22        return u"NUMERIC" 
     23     
     24    def coerce_fixedpoint_FixedPoint(self, cls, key): 
     25        warnings.warn("The precision of %s.%s cannot be determined for " 
     26                      "ODBC stores. Values may be stored incorrectly." 
     27                      % (cls.__name__, key)) 
     28        return u"NUMERIC" 
     29     
     30    def coerce_long(self, cls, key): 
     31        prop = getattr(cls, key) 
     32        bytes = int(prop.hints.get(u'bytes', 0)) 
     33        if bytes <= 4: 
     34            return self.coerce_int(cls, key) 
     35        elif bytes <= 8: 
     36            # BIGINT is usually 8 bytes 
     37            return "BIGINT" 
     38        # Anything larger than 8 bytes, use decimal/numeric. 
     39        warnings.warn("The precision of %s.%s cannot be determined for " 
     40                      "ODBC stores. Values may be stored incorrectly." 
     41                      % (cls.__name__, key)) 
     42        return u"NUMERIC" 
    2343 
    2444 
    25 class AdapterToODBCSQL(db.AdapterToSQL): 
    26     """Transform Expression values according to their type for ODBC SQL.""" 
     45class SQLDecompilerODBC(db.SQLDecompiler): 
    2746     
    28     def coerce_datetime_datetime(self, value): 
    29         return u"{ts '%s'}" % value.strftime('%Y-%m-%d %H:%M:%S') 
     47    # --------------------------- Dispatchees --------------------------- # 
    3048     
    31     def coerce_datetime_date(self, value): 
    32         return u"{d '%s'}" % value.strftime('%Y-%m-%d') 
     49    def dejavu_icontainedby(self, op1, op2): 
     50        if isinstance(op1, db.ConstWrapper): 
     51            # Looking for text in a field. Use Like (reverse terms). 
     52            return "{fn LCASE(" + op2 + ")} LIKE '%" + op1.strip("'\"").lower() + "%'" 
     53        else: 
     54            # Looking for field in (a, b, c). 
     55            # Force all args to lowercase for case-insensitive comparison. 
     56            atoms = [self.adapter.coerce(x).lower() for x in op2.basevalue] 
     57            return "{fn LCASE(%s)} IN (%s)" % (op1, ", ".join(atoms)) 
    3358     
    34     def coerce_datetime_time(self, value): 
    35         return u"{t '%s'}" % value.strftime('%H:%M:%S') 
     59    def dejavu_icontains(self, x, y): 
     60        return self.dejavu_icontainedby(y, x) 
     61     
     62    def dejavu_istartswith(self, x, y): 
     63        y = y.strip("'\"") 
     64        return "{fn LCASE(" + x + ")} LIKE '" + y + "%'" 
     65     
     66    def dejavu_iendswith(self, x, y): 
     67        y = y.strip("'\"") 
     68        return "{fn LCASE(" + x + ")} LIKE '%" + y + "'" 
     69     
     70    def dejavu_ieq(self, x, y): 
     71        return "{fn LCASE(" + x + ")} = {fn LCASE(" + y + ")}" 
     72     
     73    def dejavu_now(self): 
     74        return self.adapter.coerce(datetime.datetime.now()) 
     75     
     76    def dejavu_today(self): 
     77        return self.adapter.coerce(datetime.date.today()) 
     78     
     79    def dejavu_year(self, x): 
     80##        return "{fn DATEPART(year, " + x + ")}" 
     81        self.imperfect = True 
     82        return db.cannot_represent 
     83     
     84    def func__builtin___len(self, x): 
     85        return "{fn LENGTH(" + x + ")}" 
    3686 
    3787 
    38 dbi_datetype = type(dbi.dbiDate(0)) 
    39  
    40 class AdapterFromODBC(db.AdapterFromDB): 
    41     """Transform incoming values from ODBC to Dejavu datatypes.""" 
    42      
    43     def coerce_datetime_datetime(self, value, coltype): 
    44         if isinstance(value, dbi_datetype): 
    45             return datetime.datetime.utcfromtimestamp(int(value)) 
    46      
    47     def coerce_datetime_date(self, value, coltype): 
    48         if isinstance(value, dbi_datetype): 
    49             return datetime.datetime.utcfromtimestamp(int(value)).date() 
    50      
    51     def coerce_datetime_time(self, value, coltype): 
    52  
    53  
    54 class StoreIteratorODBC(object): 
    55     """Iterator for populating Units from storage.""" 
    56      
    57     def __init__(self, store, unitClass, expr): 
    58         self.store  = store 
    59         self.unitClass = unitClass 
    60         self.expr = expr 
    61         self.sql, self.imperfect = store.select(unitClass, expr) 
    62      
    63     def units(self): 
    64         s = self.store 
    65          
    66         res = s.execute(self.sql) 
    67         if res.num_rows(): 
    68             columns = {} 
    69             for index in xrange(res.nfields): 
    70                 columns[res.fname(index)] = (index, res.ftype(index)) 
    71              
    72             for row in xrange(res.ntuples): 
    73                 unit = self.unitClass() 
    74                 coercer = AdapterFromPg(unit) 
    75                 for key in unit.__class__.properties(): 
    76                     index, ftype = columns[key] 
    77                     value = res.getvalue(row, index) 
    78                     try: 
    79                         coercer.consume(key, value, ftype) 
    80                     except Exception, x: 
    81                         x.args += (key, ftype, value) 
    82                         raise x 
    83                 # If our SQL is imperfect, don't yield it to the 
    84                 # caller unless it passes evaluate(). 
    85                 if (not self.imperfect) or self.expr.evaluate(unit): 
    86                     yield unit 
    87         res.clear() 
    88  
    89     def load_data(self): 
    90         anRS = self.store.recordset(self.sql) 
    91         self.fieldNames = [x[0] for x in anRS.description] 
    92          
    93         for col, x in enumerate(anRS.description): 
    94             self.colIndices[x[0]] = col 
    95          
    96         self.data = anRS.fetchall() 
    97      
    98     def units(self): 
    99         self.load_data() 
    100         if len(self.data) > 0: 
    101             server = self.server 
    102             cache = server.cache(self.unitClass) 
    103             for row in self.data: 
    104                 # Notice odbc field names are lower case. 
    105                 ID = unicode(row[self.colIndices[u'id']]) 
    106                 # Search the cache to see if we've already attached this unit. 
    107                 # Use has_key() instead of 'is' or '==' because the Unit may 
    108                 # have changed its _properties since the last load. 
    109                 unit = cache['ID'].get(ID, None) 
    110                 if unit is None: 
    111                     unit = self.unitClass(server.namespace) 
    112                     self.populate_unit(unit, row) 
    113                     cache.store(unit) 
    114                 else: 
    115                     unit = unit[0] 
    116                 # If our SQL is imperfect, it's OK to ask our server 
    117                 # to accept() our new Unit, but don't yield it to the 
    118                 # caller unless it passes evaluate(). 
    119                 if (not self.imperfect) or self.expr.evaluate(unit): 
    120                     yield unit 
    121  
    122  
    123 savecoercer = AdapterToODBCSQL() 
    124 class StorageManagerODBC(storage.StorageManager): 
     88class StorageManagerODBC(db.StorageManagerDB): 
    12589    """StoreManager to save and retrieve Dejavu Units via ODBC.""" 
    12690     
    127     connection = None 
    128     prefix = 'djv' 
    129      
    130     def __init__(self, allOptions): 
    131         self.connection = None 
    132         try: 
    133             self.connect(allOptions['Connect']) 
    134         except KeyError: 
    135             pass 
    136          
    137         self.prefix = allOptions.get(u'Prefix', u"djv") 
    138      
    139     def __del__(self): 
    140         if self.connection is not None: 
    141             self.connection.close() 
    142      
    143     def connect(self, connectString): 
    144         self.connection = odbc.odbc(connectString) 
    145      
    146     def recordset(self, aQuery): 
    147         anRS = self.connection.cursor() 
    148 ##        try: 
    149         anRS.execute(aQuery) 
    150 ##        except: 
    151 ##            raise storage.StorageError(aQuery) 
    152         return anRS 
    153      
    154     def select(self, unitClass, expr): 
    155         sql = u"SELECT * FROM [%s]" % (self.prefix + safe_name(unitClass.__name__)) 
    156         w, i = self.where(expr) 
    157         sql += w 
    158         return sql, i 
    159      
    160     def where(self, expr): 
    161         atoms, i = ODBCSQLDecompiler(expr).code() 
    162         if len(atoms) > 0: 
    163             return (u" WHERE " + atoms, i) 
    164         else: 
    165             return (u"", i) 
    166      
    167     def execute(self, aQuery): 
    168         cur = self.connection.cursor() 
    169         cur.execute(aQuery) 
    170      
    171     def loader(self, server, unitClass, expr): 
    172         if unitClass.__name__ == u'UnitCollection': 
    173             aLoader = CollectionIteratorODBC 
    174             aLoader.storageManager = self 
    175         else: 
    176             aLoader = StoreIteratorODBC 
    177         return aLoader(self, unitClass, expr, server) 
    178      
    179     def save(self, unit, forceSave=False): 
    180         """Update the recordset from the Unit's data. 
    181          
    182         Notice in particular that we do not use the auto-number or 
    183         sequence generation capabilities within some databases, etc. 
    184         The ID should be already supplied by the UnitServer(s). 
    185         """ 
    186         if unit.dirty() or forceSave: 
    187             # Use an UPDATE command. 
    188             SETAtoms = [u"%s = %s" % (eachKey, savecoercer.coerce(getattr(unit, eachKey))) 
    189                         for eachKey in unit.__class__.properties()] 
    190             tablename = self.prefix + safe_name(unit.__class__.__name__) 
    191             if len(SETAtoms) > 0: 
    192                 data = self.recordset("SELECT * FROM %s WHERE ID = '%s';" 
    193                                       % (tablename, unit.ID)).fetchone() 
    194                 updateStatement = (u"UPDATE %s SET %s WHERE ID = '%s';" 
    195                                    % (tablename, u", ".join(SETAtoms), unit.ID)) 
    196                 if data: 
    197                     self.execute(updateStatement) 
    198                 else: 
    199                     # Create a row for the unit. 
    200                     # Use an INSERT (not a cursor) for better performance. 
    201                     insertStatement = (u"INSERT INTO %s (ID) VALUES ('%s');" 
    202                                        % (tablename, unit.ID)) 
    203                     self.execute(insertStatement) 
    204                     self.execute(updateStatement) 
    205             else: 
    206                 # These Units have no data other than IDs. 
    207                 # Create a row for the unit. 
    208                 # Use an INSERT (not a cursor) for better performance. 
    209                 insertStatement = (u"INSERT INTO %s%s (ID) VALUES ('%s');" 
    210                                    % (tablename, unit.ID)) 
    211                 self.execute(insertStatement) 
    212             unit.cleanse() 
    213         return True 
    214      
    215     def max_id(self, unitClass): 
    216         top1 = u"SELECT TOP 1 ID FROM [%s%s] ORDER BY Val(ID) DESC;" 
    217         recordsource = top1 % (self.prefix, safe_name(unitClass.__name__)) 
    218         anRS = self.recordset(recordsource) 
    219         data = anRS.fetchone() 
    220         if data: 
    221             fieldNames = [x[0] for x in anRS.description] 
    222             # ODBC field names are lower case. 
    223             id_index = fieldNames.index('id') 
    224             val = long(data[id_index]) 
    225             return val 
    226         else: 
    227             return 0 
    228      
    229     def _create_str_storage(unitClass, key): 
    230         """This basic string handler does not know anything about the size 
    231         limitations of the particular database. You should use one of the 
    232         subclasses for your particular database if you need storage for 
    233         strings over 255 characters.""" 
    234         prop = getattr(unitClass, key) 
    235         size = prop.hints.get(u'bytes', '255') 
    236         return u"VARCHAR(%s)" % size 
    237      
    238     createCoercions = {datetime.datetime: lambda x, y: u"TIMESTAMP", 
    239                        datetime.date: lambda x, y: u"DATE", 
    240                        datetime.time: lambda x, y: u"TIME", 
    241                        str: _create_str_storage, 
    242                        unicode: _create_str_storage, 
    243                        dict: _create_str_storage, 
    244                        fixedpoint.FixedPoint: lambda x, y: u"FLOAT", 
    245                        int: lambda x, y: u"INTEGER", 
    246                        bool: lambda x, y: u"BIT", 
    247                        } 
    248  
    249  
    250 class StorageManagerPgSQL(storage.StorageManager): 
    251     """StoreManager to save and retrieve Units via pyPgSQL 1.35.""" 
    252      
    253     decompiler = PgSQLDecompiler 
    254     createAdapter = db.FieldTypeAdapter() 
     91    identifier_caseless = True 
     92    decompiler = SQLDecompilerODBC 
     93    typeAdapter = FieldTypeAdapterODBC() 
    25594     
    25695    def __init__(self, name, arena, allOptions={}): 
    257         storage.StorageManager.__init__(self, name, arena, allOptions) 
    258          
    259         # connstring = (host=h port=p dbname=d user=u password=p options=o tty=t) 
    260         self.connstring = allOptions[u'Connect'] 
    261         atoms = self.connstring.split(" ") 
    262         for atom in atoms: 
    263             k, v = atom.split("=", 1) 
    264             setattr(self, k, v) 
    265         self.CreateIfMissing = allOptions.get(u'Create If Missing', '') 
    266         self.threaded = bool(allOptions.get(u'Threaded', '1')) 
    267         self._connection = None 
    268          
    269         self.prefix = allOptions.get(u'Prefix', u"djv") 
    270         self.reserve_lock = threading.Lock() 
    271      
    272     def identifier(self, *atoms): 
    273         ident = '"' + ''.join(map(str, atoms)).replace('"', '""') + '"' 
    274         if len(ident) > 63: 
    275             warnings.warn("Identifier is longer than 63 characters. Most " 
    276                           "installations of Postgres are limited to 63. " 
    277                           "See NAMEDATALEN.") 
    278         return ident 
    279      
    280     def shutdown(self): 
    281         if self.threaded: 
    282             t = threading.currentThread() 
    283             conn = getattr(t, "dejavu_storage_connection", None) 
    284             if conn is not None: 
    285                 conn.finish() 
    286         else: 
    287             if self._connection is not None: 
    288                 self._connection.finish() 
     96        db.StorageManagerDB.__init__(self, name, arena, allOptions) 
     97        self.connstring = allOptions['Connect'] 
    28998     
    29099    def _get_conn(self): 
    291         try: 
    292             conn = libpq.PQconnectdb(self.connstring) 
    293         except Exception, x: 
    294             if self.CreateIfMissing: 
    295                 self.create_database() 
    296                 conn = libpq.PQconnectdb(self.connstring) 
    297             else: 
    298                 raise 
    299         return conn 
    300      
    301     def connection(self): 
    302         if self.threaded: 
    303             t = threading.currentThread() 
    304             if not hasattr(t, 'dejavu_storage_connection'): 
    305                 t.dejavu_storage_connection = self._get_conn() 
    306             return t.dejavu_storage_connection 
    307         else: 
    308             if self._connection is None: 
    309                 self._connection = self._get_conn() 
    310             return self._connection 
    311      
    312     def _template_conn(self): 
    313         atoms = self.connstring.split(" ") 
    314         tmplconn = "" 
    315         for atom in atoms: 
    316             k, v = atom.split("=", 1) 
    317             if k == 'dbname': v = 'template1' 
    318             tmplconn += "%s=%s " % (k, v) 
    319         return libpq.PQconnectdb(tmplconn) 
    320      
    321     def create_database(self): 
    322         self.execute('CREATE DATABASE %s' % self.identifier(self.dbname), 
    323                      self._template_conn()) 
    324      
    325     def drop_database(self): 
    326         self.execute("DROP DATABASE %s;" % self.identifier(self.dbname), 
    327                      self._template_conn()) 
    328      
    329     def select(self, unitClass, expr, distinct_fields=None): 
    330         tablename = self.identifier(self.prefix, unitClass.__name__) 
    331         if distinct_fields: 
    332             distinct_fields = [self.identifier(x) for x in distinct_fields] 
    333             sql = (u'SELECT DISTINCT %s FROM %s' % 
    334                    (u', '.join(distinct_fields), tablename)) 
    335         else: 
    336             sql = u'SELECT * FROM %s' % tablename 
    337         w, i = self.where(unitClass, expr) 
    338         if len(w) > 0: 
    339             w = u" WHERE " + w 
    340         else: 
    341             w = u"" 
    342         sql += w 
    343         return sql, i 
    344      
    345     def where(self, cls, expr): 
    346         decom = self.decompiler(self.prefix + cls.__name__, expr) 
    347         return decom.code(), decom.imperfect 
     100        return odbc.odbc(self.connstring) 
    348101     
    349102    def execute(self, query, conn=None): 
     103        """execute(query, conn=None) -> result set.""" 
    350104        if conn is None: 
    351105            conn = self.connection() 
    352106        try: 
    353             return conn.query(query) 
     107            cursor = conn.cursor() 
     108            cursor.execute(query) 
     109            return cursor 
    354110        except Exception, x: 
    355111            x.args += (query,) 
    356112            raise x 
     113        except dbi.progError, x: 
     114            # Force query to string type 
     115            x += "\n" + str(query) 
     116            raise x 
    357117     
    358     def recall(self, cls, expr=None, pairs=None): 
    359         if expr is None: 
    360             expr = logic.Expression(lambda x: True) 
    361         return StoreIteratorPgSQL(self, cls, expr).units() 
     118    def fetch(self, query, conn=None): 
     119        """fetch(query, conn=None) -> rowdata, columns.""" 
     120        res = self.execute(query, conn) 
     121        return res.fetchall(), res.description 
    362122     
    363     def reserve(self, unit): 
    364         """reserve(unit). -> Reserve a persistent slot for unit. 
    365          
    366         Notice in particular that we do not use the auto-number or 
    367         sequence generation capabilities within some databases, etc. 
    368         The ID should be supplied by UnitSequencers via reserve(). 
    369         """ 
    370         clsname = unit.__class__.__name__ 
    371         tblname = self.identifier(self.prefix, clsname) 
    372         id = self.identifier("ID") 
    373         self.reserve_lock.acquire() 
    374         try: 
    375             if unit.ID is None: 
    376                 data = [] 
    377                 res = self.execute(u'SELECT %s FROM %s;' % (id, tblname)) 
    378                 if res.resultType != libpq.EMPTY_QUERY: 
    379                     data = [res.getvalue(row, 0) for row in xrange(res.ntuples)] 
    380                 unit.ID = unit.sequencer.next(data) 
    381              
    382             self.execute('INSERT INTO %s (%s) VALUES (%s)' % 
    383                          (tblname, id, AdapterToPgSQL.coerce(unit.ID))) 
    384         finally: 
    385             self.reserve_lock.release() 
     123    def create_database(self): 
     124        raise NotImplementedError 
    386125     
    387     def save(self, unit, forceSave=False): 
    388         """save(unit, forceSave=False) -> Update storage from unit's data.""" 
    389         if unit.dirty() or forceSave: 
    390             cls = unit.__class__ 
    391             clsname = cls.__name__ 
    392             tablename = self.identifier(self.prefix, clsname) 
    393              
    394             parms = [] 
    395             for key in cls.properties(): 
    396                 if key != "ID": 
    397                     val = AdapterToPgSQL.coerce(getattr(unit, key)) 
    398                     parms.append('%s = %s' % (self.identifier(key), val)) 
    399             sql = ('UPDATE %s SET %s WHERE %s = %s' % 
    400                    (tablename, u", ".join(parms), 
    401                     self.identifier("ID"), 
    402                     AdapterToPgSQL.coerce(unit.ID, cls.property_type("ID")))) 
    403             self.execute(sql) 
    404             unit.cleanse() 
    405      
    406     def destroy(self, unit): 
    407         """Delete the unit.""" 
    408         # Use a DELETE command instead of a cursor for better performance. 
    409         deleteStatement = (u'DELETE * FROM %s WHERE %s = %s' % 
    410                            (self.identifier(self.prefix, unit.__class__.__name__), 
    411                             self.identifier("ID"), 
    412                             AdapterToPgSQL.coerce(unit.ID))) 
    413         self.execute(deleteStatement) 
    414      
    415     def create_storage(self, unitClass): 
    416         tblname = self.identifier(self.prefix, unitClass.__name__) 
    417          
    418         coerce = self.createAdapter.coerce 
    419         fields = [] 
    420         for key in unitClass.properties(): 
    421             fields.append(u'%s %s' % (self.identifier(key), 
    422                                       coerce(unitClass, key))) 
    423         try: 
    424             self.execute(u'CREATE TABLE %s (%s)' % (tblname, ", ".join(fields))) 
    425         except libpq.OperationalError, x: 
    426             if not x.args[0].endswith(' already exists\n'): 
    427                 raise 
    428         else: 
    429             for index in unitClass.indices(): 
    430                 indexname = self.identifier(self.prefix, "i", 
    431                                             unitClass.__name__, index) 
    432                 self.execute(u'CREATE INDEX %s ON %s (%s)' 
    433                              % (indexname, tblname, self.identifier(index))) 
    434      
    435     def distinct(self, cls, fields, expr=None): 
    436         """Return distinct values for specified fields.""" 
    437         if expr is None: 
    438             expr = logic.Expression(lambda x: True) 
    439          
    440         # ^%$#@! There's no way to handle imperfect queries without 
    441         # creating all involved Units, which defeats the purpose of 
    442         # distinct, which was a speed issue more than anything. Grr. 
    443         sql, imperfect = self.select(cls, expr, fields) 
    444         # Ignore for now. 
    445 ##        if imperfect: 
    446 ##            raise ValueError(u"The following query cannot be reliably " 
    447 ##                             u"returned from a Postgres data source.", 
    448 ##                             u"distinct()", cls, fields, expr) 
    449          
    450         res = self.execute(sql) 
    451         if res.resultType == libpq.EMPTY_QUERY: 
    452             return [] 
    453          
    454         coerce = AdapterFromPg().coerce 
    455         data = [] 
    456         for row in xrange(res.ntuples): 
    457             coerced_row = [] 
    458             for i in xrange(len(fields)): 
    459                 expectedType = cls.property_type(field[i]) 
    460                 actualType = res.ftype(i) 
    461                 val = coerce(res.getvalue(row, i), actualType, expectedType) 
    462                 coerced_row.append(val) 
    463             data.append(coerced_row) 
    464         return zip(*data) 
     126    def drop_database(self): 
     127        raise NotImplementedError 
    465128 

  • trunk/storage/storepypgsql.py

    r45 r46  
    11# Use libpq directly to avoid all of the DB-API overhead. 
    22from pyPgSQL import libpq 
    3  
    4 import warnings 
    5 import threading 
    63import datetime 
    7  
    84import dejavu 
    9 from dejavu import storage, logic 
    105from dejavu.storage import db 
    11  
    12  
    13 AdapterToPgSQL = db.AdapterToSQL() 
    14 AdapterFromPg = db.AdapterFromDB 
    15  
    16  
    17 class StoreIteratorPgSQL(object): 
    18     """Iterator for populating Units from storage.""" 
    19      
    20     def __init__(self, store, unitClass, expr): 
    21         self.store  = store 
    22         self.unitClass = unitClass 
    23         self.expr = expr 
    24         self.sql, self.imperfect = store.select(unitClass, expr) 
    25      
    26     def units(self): 
    27         s = self.store 
    28          
    29         res = s.execute(self.sql) 
    30         if res.resultType != libpq.EMPTY_QUERY: 
    31             columns = {} 
    32             for index in xrange(res.nfields): 
    33                 columns[res.fname(index)] = (index, res.ftype(index)) 
    34              
    35             for row in xrange(res.ntuples): 
    36                 unit = self.unitClass() 
    37                 coercer = AdapterFromPg(unit) 
    38                 for key in unit.__class__.properties(): 
    39                     index, ftype = columns[key] 
    40                     value = res.getvalue(row, index) 
    41                     try: 
    42                         coercer.consume(key, value, ftype) 
    43                     except Exception, x: 
    44                         x.args += (key, ftype, value) 
    45                         raise x 
    46                 # If our SQL is imperfect, don't yield it to the 
    47                 # caller unless it passes evaluate(). 
    48                 if (not self.imperfect) or self.expr.evaluate(unit): 
    49                     yield unit 
    50         res.clear() 
    516 
    527 
    …… 
    7833 
    7934 
    80 class StorageManagerPgSQL(storage.StorageManager): 
     35class StorageManagerPgSQL(db.StorageManagerDB): 
    8136    """StoreManager to save and retrieve Units via pyPgSQL 1.35.""" 
    8237     
     38    identifier_length = 63 
     39    close_connection_method = 'finish' 
    8340    decompiler = PgSQLDecompiler 
    84     createAdapter = db.FieldTypeAdapter() 
    8541     
    8642    def __init__(self, name, arena, allOptions={}): 
    87         storage.StorageManager.__init__(self, name, arena, allOptions) 
     43        db.StorageManagerDB.__init__(self, name, arena, allOptions) 
    8844         
    8945        # connstring = (host=h port=p dbname=d user=u password=p options=o tty=t) 
    …… 
    9349            k, v = atom.split("=", 1) 
    9450            setattr(self, k, v) 
    95         self.CreateIfMissing = allOptions.get(u'Create If Missing', '') 
    96         self.threaded = bool(allOptions.get(u'Threaded', '1')) 
    97         self._connection = None 
    98          
    99         self.prefix = allOptions.get(u'Prefix', u"djv") 
    100         self.reserve_lock = threading.Lock() 
    101      
    102     def identifier(self, *atoms): 
    103         ident = '"' + ''.join(map(str, atoms)).replace('"', '""') + '"' 
    104         if len(ident) > 63: 
    105             warnings.warn("Identifier is longer than 63 characters. Most " 
    106                           "installations of Postgres are limited to 63. " 
    107                           "See NAMEDATALEN.") 
    108         return ident 
    109      
    110     def shutdown(self): 
    111         if self.threaded: 
    112             t = threading.currentThread() 
    113             conn = getattr(t, "dejavu_storage_connection", None) 
    114             if conn is not None: 
    115                 conn.finish() 
    116         else: 
    117             if self._connection is not None: 
    118                 self._connection.finish() 
    11951     
    12052    def _get_conn(self): 
    …… 
    12860                raise 
    12961        return conn 
    130      
    131     def connection(self): 
    132         if self.threaded: 
    133             t = threading.currentThread() 
    134             if not hasattr(t, 'dejavu_storage_connection'): 
    135                 t.dejavu_storage_connection = self._get_conn() 
    136             return t.dejavu_storage_connection 
    137         else: 
    138             if self._connection is None: 
    139                 self._connection = self._get_conn() 
    140             return self._connection 
    14162     
    14263    def _template_conn(self): 
    …… 
    15778                     self._template_conn()) 
    15879     
    159     def select(self, unitClass, expr, distinct_fields=None): 
    160         tablename = self.identifier(self.prefix, unitClass.__name__) 
    161         if distinct_fields: 
    162             distinct_fields = [self.identifier(x) for x in distinct_fields] 
    163             sql = (u'SELECT DISTINCT %s FROM %s' % 
    164                    (u', '.join(distinct_fields), tablename)) 
    165         else: 
    166             sql = u'SELECT * FROM %s' % tablename 
    167         w, i = self.where(unitClass, expr) 
    168         if len(w) > 0: 
    169             w = u" WHERE " + w 
    170         else: 
    171             w = u"" 
    172         sql += w 
    173         return sql, i 
    174      
    175     def where(self, cls, expr): 
    176         decom = self.decompiler(self.prefix + cls.__name__, expr) 
    177         return decom.code(), decom.imperfect 
    178      
    179     def execute(self, query, conn=None): 
    180         if conn is None: 
    181             conn = self.connection() 
    182         try: 
    183             return conn.query(query) 
    184         except Exception, x: 
    185             x.args += (query,) 
    186             raise x 
    187      
    188     def recall(self, cls, expr=None, pairs=None): 
    189         if expr is None: 
    190             expr = logic.Expression(lambda x: True) 
    191         return StoreIteratorPgSQL(self, cls, expr).units() 
    192      
    193     def reserve(self, unit): 
    194         """reserve(unit). -> Reserve a persistent slot for unit. 
     80    def fetch(self, query, conn=None): 
     81        """fetch(query, conn=None) -> rowdata, columns.""" 
     82        res = self.execute(query, conn) 
    19583         
    196         Notice in particular that we do not use the auto-number or 
    197         sequence generation capabilities within some databases, etc. 
    198         The ID should be supplied by UnitSequencers via reserve(). 
    199         """ 
    200         clsname = unit.__class__.__name__ 
    201         tblname = self.identifier(self.prefix, clsname) 
    202         id = self.identifier("ID") 
    203         self.reserve_lock.acquire() 
    204         try: 
    205             if unit.ID is None: 
    206                 data = [] 
    207                 res = self.execute(u'SELECT %s FROM %s;' % (id, tblname)) 
    208                 if res.resultType != libpq.EMPTY_QUERY: 
    209                     data = [res.getvalue(row, 0) for row in xrange(res.ntuples)] 
    210                 unit.ID = unit.sequencer.next(data) 
    211              
    212             self.execute('INSERT INTO %s (%s) VALUES (%s)' % 
    213                          (tblname, id, AdapterToPgSQL.coerce(unit.ID))) 
    214         finally: 
    215             self.reserve_lock.release() 
    216      
    217     def save(self, unit, forceSave=False): 
    218         """save(unit, forceSave=False) -> Update storage from unit's data.""" 
    219         if unit.dirty() or forceSave: 
    220             cls = unit.__class__ 
    221             clsname = cls.__name__ 
    222             tablename = self.identifier(self.prefix, clsname) 
    223              
    224             parms = [] 
    225             for key in cls.properties(): 
    226                 if key != "ID": 
    227                     val = AdapterToPgSQL.coerce(getattr(unit, key)) 
    228                     parms.append('%s = %s' % (self.identifier(key), val)) 
    229             sql = ('UPDATE %s SET %s WHERE %s = %s' % 
    230                    (tablename, u", ".join(parms), 
    231                     self.identifier("ID"), 
    232                     AdapterToPgSQL.coerce(unit.ID, cls.property_type("ID")))) 
    233             self.execute(sql) 
    234             unit.cleanse() 
    235      
    236     def destroy(self, unit): 
    237         """Delete the unit.""" 
    238         # Use a DELETE command instead of a cursor for better performance. 
    239         deleteStatement = (u'DELETE * FROM %s WHERE %s = %s' % 
    240                            (self.identifier(self.prefix, unit.__class__.__name__), 
    241                             self.identifier("ID"), 
    242                             AdapterToPgSQL.coerce(unit.ID))) 
    243         self.execute(deleteStatement) 
    244      
    245     def create_storage(self, unitClass): 
    246         tblname = self.identifier(self.prefix, unitClass.__name__) 
     84        columns = [] 
     85        if res.resultType != libpq.EMPTY_QUERY: 
     86            for index in xrange(res.nfields): 
     87                columns.append((res.fname(index), res.ftype(index))) 
    24788         
    248         coerce = self.createAdapter.coerce 
    249         fields = [] 
    250         for key in unitClass.properties(): 
    251             fields.append(u'%s %s' % (self.identifier(key), 
    252                                       coerce(unitClass, key))) 
    253         try: 
    254             self.execute(u'CREATE TABLE %s (%s)' % (tblname, ", ".join(fields))) 
    255         except libpq.OperationalError, x: 
    256             if not x.args[0].endswith(' already exists\n'): 
    257                 raise 
    258         else: 
    259             for index in unitClass.indices(): 
    260                 indexname = self.identifier(self.prefix, "i", 
    261                                             unitClass.__name__, index) 
    262                 self.execute(u'CREATE INDEX %s ON %s (%s)' 
    263                              % (indexname, tblname, self.identifier(index))) 
    264      
    265     def distinct(self, cls, fields, expr=None): 
    266         """Return distinct values for specified fields.""" 
    267         if expr is None: 
    268             expr = logic.Expression(lambda x: True) 
     89        def iterator(): 
     90            for row in xrange(res.ntuples): 
     91                yield [res.getvalue(row, col) for col in xrange(res.nfields)] 
     92            # This should be more robust--needs a class with a cleanup call. 
     93            res.clear() 
    26994         
    270         # ^%$#@! There's no way to handle imperfect queries without 
    271         # creating all involved Units, which defeats the purpose of 
    272         # distinct, which was a speed issue more than anything. Grr. 
    273         sql, imperfect = self.select(cls, expr, fields) 
    274         # Ignore for now. 
    275 ##        if imperfect: 
    276 ##            raise ValueError(u"The following query cannot be reliably " 
    277 ##                             u"returned from a Postgres data source.", 
    278 ##                             u"distinct()", cls, fields, expr) 
    279          
    280         res = self.execute(sql) 
    281         if res.resultType == libpq.EMPTY_QUERY: 
    282             return [] 
    283          
    284         coerce = AdapterFromPg().coerce 
    285         data = [] 
    286         for row in xrange(res.ntuples): 
    287             coerced_row = [] 
    288             for i in xrange(len(fields)): 
    289                 expectedType = cls.property_type(field[i]) 
    290                 actualType = res.ftype(i) 
    291                 val = coerce(res.getvalue(row, i), actualType, expectedType) 
    292                 coerced_row.append(val) 
    293             data.append(coerced_row) 
    294         return zip(*data) 
     95        return iterator(), columns 
    29596 

  • trunk/storage/storeshelve.py

    r45 r46  
    4646                    unit._properties = unitdict 
    4747                    if expr is None or expr.evaluate(unit): 
     48                        unit.cleanse() 
    4849                        units.append(unit) 
    4950        finally: 
    …… 
    103104        try: 
    104105            globs = {} 
    105             for unit in data.itervalues(): 
     106            for unitdict in data.itervalues(): 
    106107                unit = cls() 
    107                 # Set the attribute directly to avoid __set__ overhead. 
     108                # Set the attributes directly to avoid __set__ overhead. 
    108109                unit._properties = unitdict 
    109110                if expr is None or expr.evaluate(unit): 
    110111                    key = tuple([getattr(unit, field) for field in fields]) 
    111                     globs[tuple] = None 
     112                    globs[key] = None 
    112113            return globs.keys() 
    113114        finally: 

  • trunk/storage/storesqlite.py

    r45 r46  
    1010 
    1111 
    12 AdapterFromSQLite = db.AdapterFromDB 
    13  
    14 class _AdapterToSQLite(db.AdapterToSQL): 
     12class AdapterToSQLite(db.AdapterToSQL): 
    1513     
    1614    def coerce_bool(self, value): 
    …… 
    1917        return '0' 
    2018 
    21 AdapterToSQLite = _AdapterToSQLite() 
    22  
    2319 
    2420class SQLiteDecompiler(db.SQLDecompiler): 
    2521     
    2622    def column_name(self, name): 
    27         return '[%s].[%s]' % (self.tablename, name) 
    28 ##     
    29 ##    # --------------------------- Dispatchees --------------------------- # 
     23        return '%s.[%s]' % (self.tablename, name) 
     24     
     25    # --------------------------- Dispatchees --------------------------- # 
    3026     
    3127    def dejavu_now(self): 
    …… 
    4036 
    4137 
    42 class StoreIteratorSQLite(object): 
    43     """Iterator for populating Units from storage.""" 
     38class StorageManagerSQLite(db.StorageManagerDB): 
     39    """StoreManager to save and retrieve Units via _sqlite.""" 
    4440     
    45     def __init__(self, store, unitClass, expr): 
    46         self.store  = store 
    47         self.unitClass = unitClass 
    48         self.expr = expr 
    49         self.sql, self.imperfect = store.select(unitClass, expr) 
    50      
    51     def units(self): 
    52         s = self.store 
    53          
    54         res = s.execute(self.sql) 
    55         if len(res.row_list): 
    56             columns = {} 
    57             for index, col in enumerate(res.col_defs): 
    58                 # name, type_code, display_size, internal_size, precision, scale, null_ok 
    59                 columns[col[0]] = (index, col[1]) 
    60              
    61             for row in res.row_list: 
    62                 unit = self.unitClass() 
    63                 coercer = AdapterFromSQLite(unit) 
    64                 for key in unit.__class__.properties(): 
    65                     index, ftype = columns[key] 
    66                     value = row[index] 
    67                     try: 
    68                         coercer.consume(key, value, ftype) 
    69                     except Exception, x: 
    70                         x.args += (key, value, ftype) 
    71                         raise x 
    72                 # If our SQL is imperfect, don't yield it to the 
    73                 # caller unless it passes evaluate(). 
    74                 if (not self.imperfect) or self.expr.evaluate(unit): 
    75                     yield unit 
    76  
    77  
    78 class StorageManagerSQLite(storage.StorageManager): 
    79     """StoreManager to save and retrieve Units via _SQLite .""" 
    80      
    81     threaded = False 
     41    identifier_length = 0 
     42    decompiler = SQLiteDecompiler 
     43    toAdapter = AdapterToSQLite() 
    8244     
    8345    def __init__(self, name, arena, allOptions={}): 
    84         storage.StorageManager.__init__(self, name, arena, allOptions) 
    85          
    86         self.CreateIfMissing = allOptions.get(u'Create If Missing', '') 
    87         self.threaded = bool(allOptions.get(u'Threaded', '1')) 
    88         self._connection = None 
     46        db.StorageManagerDB.__init__(self, name, arena, allOptions) 
    8947         
    9048        self.database = allOptions.get(u'Database', '') 
    9149        self.mode = int(allOptions.get(u'Mode', '0755'), 8) 
    92          
    93         self.prefix = allOptions.get(u'Prefix', u"djv") 
    94         self.reserve_lock = threading.Lock() 
    95          
    96         self.decompiler = SQLiteDecompiler 
    9750     
    9851    def identifier(self, *atoms): 
    …… 
    11063        ...we'll use the third option (square brackets). 
    11164        """ 
    112         ident = "[" + ''.join(map(str, atoms)) + "]" 
    113         return ident 
    114      
    115     def shutdown(self): 
    116         if self.threaded: 
    117             t = threading.currentThread() 
    118             conn = getattr(t, "dejavu_storage_connection", None) 
    119             if conn is not None: 
    120                 conn.close() 
    121         else: 
    122             if self._connection is not None: 
    123                 self._connection.close() 
     65        return "[" + ''.join(map(str, atoms)) + "]" 
    12466     
    12567    def _get_conn(self): 
    …… 
    12769        return _sqlite.connect(self.database, self.mode) 
    12870     
    129     def connection(self): 
    130         if self.threaded: 
    131             t = threading.currentThread() 
    132             if not hasattr(t, 'dejavu_storage_connection'): 
    133                 t.dejavu_storage_connection = self._get_conn() 
    134             return t.dejavu_storage_connection 
    135         else: 
    136             if self._connection is None: 
    137                 self._connection = self._get_conn() 
    138             return self._connection 
     71    create_database = _get_conn 
    13972     
    14073    def drop_database(self): 
    …… 
    14376        os.remove(self.database) 
    14477     
    145     def select(self, unitClass, expr, distinct_fields=None): 
    146         tablename = self.identifier(self.prefix, unitClass.__name__) 
    147         if distinct_fields: 
    148             distinct_fields = [self.identifier(x) for x in distinct_fields] 
    149             sql = (u'SELECT DISTINCT %s FROM %s' % 
    150                    (u', '.join(distinct_fields), tablename)) 
    151         else: 
    152             sql = u'SELECT * FROM %s' % tablename 
    153         w, i = self.where(unitClass, expr) 
    154         if len(w) > 0: 
    155             w = u" WHERE " + w 
    156         else: 
    157             w = u"" 
    158         sql += w 
    159         return sql, i 
    160      
    161     def where(self, cls, expr): 
    162         tablename = self.prefix + cls.__name__ 
    163         decom = self.decompiler(tablename, expr, _AdapterToSQLite) 
    164         return decom.code(), decom.imperfect 
    165      
    16678    def execute(self, query, conn=None): 
    16779        if conn is None: 
    …… 
    16981        try: 
    17082            return conn.execute(query) 
     83            #           ^^^^^^^ 
    17184        except Exception, x: 
    17285            x.args += (query,) 
    17386            raise x 
    17487     
    175     def recall(self, cls, expr=None, pairs=None): 
    176         if expr is None: 
    177             expr = logic.Expression(lambda x: True) 
    178         return StoreIteratorSQLite(self, cls, expr).units() 
    179      
    180     def reserve(self, unit): 
    181         """reserve(unit). -> Reserve a persistent slot for unit. 
    182          
    183         Notice in particular that we do not use the auto-number or 
    184         sequence generation capabilities within some databases, etc. 
    185         The ID should be supplied by UnitSequencers via reserve(). 
    186         """ 
    187         clsname = unit.__class__.__name__ 
    188         tblname = self.identifier(self.prefix, clsname) 
    189         id = self.identifier("ID") 
    190         self.reserve_lock.acquire() 
    191         try: 
    192             coerce = AdapterFromSQLite(unit).coerce 
    193             expectedType = unit.__class__.property_type("ID") 
    194             if unit.ID is None: 
    195                 data = [] 
    196                 res = self.execute(u'SELECT %s FROM %s;' % (id, tblname)) 
    197                 if len(res.row_list): 
    198                     data = [coerce(x[0], type(x[0]), expectedType) 
    199                             for x in res.row_list] 
    200                 unit.ID = unit.sequencer.next(data) 
    201              
    202             self.execute('INSERT INTO %s (%s) VALUES (%s);' % 
    203                          (tblname, id, AdapterToSQLite.coerce(unit.ID))) 
    204         finally: 
    205             self.reserve_lock.release() 
    206      
    207     def save(self, unit, forceSave=False): 
    208         """save(unit, forceSave=False) -> Update storage from unit's data.""" 
    209         if unit.dirty() or forceSave: 
    210             cls = unit.__class__ 
    211             tablename = self.identifier(self.prefix, cls.__name__) 
    212              
    213             parms = [] 
    214             for key in cls.properties(): 
    215                 if key != "ID": 
    216                     val = AdapterToSQLite.coerce(getattr(unit, key)) 
    217                     parms.append('%s = %s' % (self.identifier(key), val)) 
    218             sql = ('UPDATE %s SET %s WHERE %s = %s' % 
    219                    (tablename, u", ".join(parms), self.identifier("ID"), 
    220                     AdapterToSQLite.coerce(unit.ID, cls.property_type("ID")))) 
    221             self.execute(sql) 
    222             unit.cleanse() 
    223      
    224     def destroy(self, unit): 
    225         """Delete the unit.""" 
    226         # Use a DELETE command instead of a cursor for better performance. 
    227         deleteStatement = (u'DELETE FROM %s WHERE %s = %s' % 
    228                            (self.identifier(self.prefix, unit.__class__.__name__), 
    229                             self.identifier("ID"), 
    230                             AdapterToSQLite.coerce(unit.ID))) 
    231         self.execute(deleteStatement) 
    232      
    23388    def create_storage(self, unitClass): 
    234         clsname = unitClass.__name__ 
    235         tblname = self.identifier(self.prefix, clsname) 
     89        tablename = self.tablename(unitClass) 
    23690         
    23791        # SQLite is typeless. 
    23892        fields = [self.identifier(key) for key in unitClass.properties()] 
    23993         
    240 ##        try: 
    241         self.execute(u'CREATE TABLE %s (%s)' % (tblname, ", ".join(fields))) 
    242 ##        except Exception, x: 
    243 ##            if not x.args[1].endswith(" already exists"): 
    244 ##                raise 
    245 ##        else: 
     94        self.execute(u'CREATE TABLE %s (%s);' % (tablename, ", ".join(fields))) 
    24695        for index in unitClass.indices(): 
    247             indexname = self.identifier(self.prefix, "i", clsname, index) 
    248             self.execute(u'CREATE INDEX %s ON %s (%s)' 
    249                          % (indexname, tblname, self.identifier(index))) 
    250      
    251     def distinct(self, cls, fields, expr=None): 
    252         """Return distinct values for specified fields.""" 
    253         if expr is None: 
    254             expr = logic.Expression(lambda x: True) 
    255          
    256         # ^%$#@! There's no way to handle imperfect queries without 
    257         # creating all involved Units, which defeats the purpose of 
    258         # distinct, which was a speed issue more than anything. Grr. 
    259         sql, imperfect = self.select(cls, expr, fields) 
    260         # Ignore for now. 
    261 ##        if imperfect: 
    262 ##            raise ValueError(u"The following query cannot be reliably " 
    263 ##                             u"returned from a Postgres data source.", 
    264 ##                             u"distinct()", cls, fields, expr) 
    265          
    266         res = self.execute(sql) 
    267         if res.num_rows(): 
    268             data = [] 
    269             coerce = AdapterFromSQLite().coerce 
    270             for row in data: 
    271                 coerced_row = [] 
    272                 for i in xrange(len(fields)): 
    273                     expectedType = cls.property_type(field[i]) 
    274                     actualType = res.ftype(i) 
    275                     val = coerce(res.getvalue(row, i), actualType, expectedType) 
    276                     coerced_row.append(val) 
    277                 data.append(coerced_row) 
    278             return zip(*data) 
    279         else: 
    280             return [] 
     96            i = self.identifier(self.prefix, "i", unitClass.__name__, index) 
     97            self.execute(u'CREATE INDEX %s ON %s (%s);' % 
     98                         (i, tablename, self.identifier(index))) 
    28199 

  • trunk/storage/test_storeado.py

    r45 r46  
    55    # Microsoft Access 
    66    opts = {u'Connect': "PROVIDER=MICROSOFT.JET.OLEDB.4.0;DATA SOURCE=zoo.mdb;", 
    7             u'Expanded Columns': "Animal.PreviousZoos", 
     7            u'Expanded Columns': "Animal.PreviousZoos:int", 
    88            u'Create If Missing': True, 
    99            } 
    1010 
    1111    try: 
    12         testSM = zoo_fixture.setup_SM("dejavu.storage.storeado.StorageManagerADO_MSAccess", opts) 
     12        zoo_fixture.setup_SM("dejavu.storage.storeado.StorageManagerADO_MSAccess", opts) 
    1313        zoo_fixture.run_tests() 
    1414    finally: 
    15         zoo_fixture.zoo.arena.shutdown() 
    16         testSM.drop_database() 
     15        zoo_fixture.teardown() 

  • trunk/storage/test_storemysql.py

    r45 r46  
    22 
    33if __name__ == '__main__': 
    4     dbname = raw_input("Database name [dejavu_test]:") or "dejavu_test" 
    54    pword = raw_input("Password for the root user:") 
    65    opts = {"host": "localhost", 
    7             "db": dbname, 
     6            "db": "dejavu_test", 
    87            "user": "root", 
    98            "passwd": pword, 
    …… 
    1211     
    1312    try: 
    14         testSM = zoo_fixture.setup_SM("dejavu.storage.storemysql.StorageManagerMySQL", opts) 
     13        zoo_fixture.setup_SM("dejavu.storage.storemysql.StorageManagerMySQL", opts) 
    1514        zoo_fixture.run_tests() 
    1615    finally: 
    17         try: 
    18             testSM.drop_database() 
    19         except NameError: 
    20             pass 
    21         zoo_fixture.zoo.arena.shutdown() 
     16        zoo_fixture.teardown() 

  • trunk/storage/test_storeodbc.py

    r45 r46  
    11from dejavu.storage import zoo_fixture 
    22 
     3 
     4def create_access_db(): 
     5    # Create the database. 
     6    import win32com.client 
     7    cat = win32com.client.Dispatch(r'ADOX.Catalog') 
     8    cat.Create("PROVIDER=MICROSOFT.JET.OLEDB.4.0;DATA SOURCE=zoo.mdb;") 
     9 
     10 
    311if __name__ == '__main__': 
     12    create_access_db() 
     13     
    414    # Once again, we find that the first param must be repeated 
    515    # in the connection string. Not sure why. 
    …… 
    717                         "Driver={Microsoft Access Driver (*.mdb)};" 
    818                         "DBQ=zoo.mdb;Provider=MSDASQL;"), 
     19            u'Expanded Columns': "Animal.PreviousZoos:int", 
    920            } 
    1021     
    11     # Create the database. 
    12     import win32com.client 
    13     cat = win32com.client.Dispatch(r'ADOX.Catalog') 
    14     cat.Create(opts['Connect']) 
    15      
    1622    try: 
    17         testSM = zoo_fixture.setup_SM("dejavu.storage.storeodbc.StorageManagerODBC", opts) 
     23        zoo_fixture.setup_SM("dejavu.storage.storeodbc.StorageManagerODBC", opts) 
    1824        zoo_fixture.run_tests() 
    1925    finally: 
    20         zoo_fixture.zoo.arena.shutdown() 
     26        zoo_fixture.teardown() 
    2127        try: 
    22             testSM.drop_database() 
    23         except NameError: 
    24             pass 
     28            import os; os.remove("zoo.mdb") 
     29        except OSError: 
     30            print "Could not remove database." 

  • trunk/storage/test_storepypgsql.py

    r45 r46  
    22 
    33if __name__ == '__main__': 
    4     dbname = raw_input("Database name [dejavu_test]:") or "dejavu_test" 
    54    pword = raw_input("Password for the postgres user:") 
    6     opts = {u'Connect': ("host=localhost dbname=%s user=postgres password=%s" 
    7                          % (dbname, pword)), 
     5    opts = {u'Connect': ("host=localhost dbname=dejavu_test " 
     6                         "user=postgres password=%s" % pword), 
    87            u'Create If Missing': True, 
    98            } 
    109     
    1110    try: 
    12         testSM = zoo_fixture.setup_SM("dejavu.storage.storepypgsql.StorageManagerPgSQL", opts) 
     11        zoo_fixture.setup_SM("dejavu.storage.storepypgsql.StorageManagerPgSQL", opts) 
    1312        zoo_fixture.run_tests() 
    1413    finally: 
    15         zoo_fixture.zoo.arena.shutdown() 
    16         try: 
    17             testSM.drop_database() 
    18         except NameError: 
    19             pass 
     14        zoo_fixture.teardown() 
    2015 

  • trunk/storage/test_storeshelve.py

    r45 r46  
    1313     
    1414    try: 
    15         testSM = zoo_fixture.setup_SM("dejavu.storage.storeshelve.StorageManagerShelve", opts) 
     15        zoo_fixture.setup_SM("dejavu.storage.storeshelve.StorageManagerShelve", opts) 
    1616        zoo_fixture.run_tests() 
    1717    finally: 
    18         zoo_fixture.zoo.arena.shutdown() 
    19         try: 
    20             testSM.drop_database() 
    21         except NameError: 
    22             pass 
     18        zoo_fixture.teardown() 

  • trunk/storage/test_storesqlite.py

    r45 r46  
    55     
    66    try: 
    7         testSM = zoo_fixture.setup_SM("dejavu.storage.storesqlite.StorageManagerSQLite", opts) 
     7        zoo_fixture.setup_SM("dejavu.storage.storesqlite.StorageManagerSQLite", opts) 
    88        zoo_fixture.run_tests() 
    99    finally: 
    10         zoo_fixture.zoo.arena.shutdown() 
    11         try: 
    12             testSM.drop_database() 
    13         except NameError: 
    14             pass 
     10        zoo_fixture.teardown() 

  • trunk/storage/zoo_fixture.py

    r45 r46  
    33Don't run this directly; call it from a script for a specific 
    44Storage Manager, setting up and tearing down that SM as needed. 
     5See test_store*.py in this directory. 
    56""" 
    67 
    …… 
    1314class ZooTests(unittest.TestCase): 
    1415     
    15     def setUp(self): 
     16    def test_0_populate(self): 
    1617        box = zoo.arena.new_sandbox() 
    1718         
    …… 
    2021        WAP = zoo.Zoo(Name = 'Wild Animal Park', 
    2122                      Founded = datetime.date(2000, 1, 1), 
    22                       # 59 should give rounding errors with divmod, which 
    23                       # AdapterFromDB needs to correct. 
     23                      # 59 can give rounding errors with divmod, which 
     24                      # AdapterFromADO needs to correct. 
    2425                      Opens = datetime.time(8, 15, 59), 
    2526                      LastEscape = datetime.datetime(2004, 7, 29, 5, 6, 7), 
     27                      Admission = "4.95", 
    2628                      ) 
    2729        box.memorize(WAP) 
    2830         
    2931        SDZ = zoo.Zoo(Name = 'San Diego Zoo', 
     32                      # This early date should play havoc with a number 
     33                      # of implementations. 
    3034                      Founded = datetime.date(1835, 9, 13), 
    3135                      Opens = datetime.time(9, 0, 0), 
     36                      Admission = "0", 
    3237                      ) 
    3338        box.memorize(SDZ) 
    …… 
    5459        box.flush_all() 
    5560     
    56     def test_Object_data(self): 
    57         """Test zoo objects one by one.""" 
     61    def test_1_Object_Properties(self): 
    5862        box = zoo.arena.new_sandbox() 
    5963         
    …… 
    6569        self.assertEqual(WAP.LastEscape, 
    6670                         datetime.datetime(2004, 12, 21, 8, 15, 0)) 
     71        self.assertEqual(str(WAP.Admission), "4.95") 
    6772         
    6873        SDZ = box.unit(zoo.Zoo, Founded=datetime.date(1835, 9, 13)) 
    …… 
    7176        self.assertEqual(SDZ.Opens, datetime.time(9, 0, 0)) 
    7277        self.assertEqual(SDZ.LastEscape, None) 
     78        self.assertEqual(float(SDZ.Admission), 0) 
    7379         
    7480        leopard = box.unit(zoo.Animal, Name='Leopard') 
    …… 
    94100        self.assertEqual(millipede.LastEscape, None) 
    95101     
    96     def test_Expressions(self): 
     102    def test_2_Expressions(self): 
    97103        box = zoo.arena.new_sandbox() 
    98104         
    …… 
    103109            return len([x for x in units]) 
    104110         
    105         self.assertEqual(len([x for x in box.recall(zoo.Zoo)]), 2) 
     111        zoos = [x for x in box.recall(zoo.Zoo)] 
     112        self.assertEqual(zoos[0].dirty(), False) 
     113        self.assertEqual(len(zoos), 2) 
    106114        self.assertEqual(matches(lambda x: True), 8) 
    107115        self.assertEqual(matches(lambda x: x.Legs == 4), 4) 
    …… 
    147155        self.assertEqual(matches(lambda x: 'p' in x.Name 
    148156                                 and x.Name.count('e') > 1), 2) 
     157     
     158    def test_3_Aggregates(self): 
     159        box = zoo.arena.new_sandbox() 
     160        legs = box.distinct(zoo.Animal, ['Legs']) 
     161        legs.sort() 
     162        self.assertEqual(legs, [1, 2, 4, 100, 1000000]) 
     163 
    149164 
    150165def setup_SM(SM_class, opts): 
    …… 
    157172     
    158173    for cls in (zoo.Animal, zoo.Zoo, zoo.Exhibit): 
     174        zoo.arena.create_storage(cls) 
     175 
     176def teardown(): 
     177    zoo.arena.shutdown() 
     178    for store in zoo.arena.stores.values(): 
    159179        try: 
    160             zoo.arena.create_storage(cls) 
    161             box = zoo.arena.new_sandbox() 
    162             for unit in box.recall(cls): 
    163                 unit.forget() 
    164         except: 
     180            store.drop_database() 
     181        except NotImplementedError: 
    165182            pass 
    166      
    167     return testSM 
    168183 
    169184def run_tests(): 

  • trunk/zoo.py

    r45 r46  
    11import datetime 
     2try: 
     3    import fixedpoint 
     4except ImportError: 
     5    fixedpoint = None 
     6 
     7try: 
     8    import decimal 
     9except ImportError: 
     10    decimal = None 
     11 
    212import dejavu 
    313from dejavu import Unit, UnitProperty, associate 
    …… 
    919    Opens = UnitProperty(datetime.time) 
    1020    LastEscape = UnitProperty(datetime.datetime) 
     21     
     22    if fixedpoint: 
     23        Admission = UnitProperty(fixedpoint.FixedPoint) 
     24    else: 
     25        Admission = UnitProperty(float) 
    1126 
    1227