Tutorial¶
First steps¶
You must add denorm app to your INSTALLED_APPS within your settings.py:
INSTALLED_APPS = (
...
'denorm',
...
)
You also need to run the initial migration for denorm app:
$ python manage.py migrate denorm
Creating denormalized fields using callback functions¶
A denormalized field can be created from a python function by using the @denormalized
decorator.
The decorator takes at least one argument: the database field type you want to use to store the computed
value. Any additional arguments will be passed into the constructor of the specified field when it is actually
created.
If you already use the @property
decorator that comes with python to make your computed values accessible
like attributes, you can often just replace @property
with @denormalized(..)
and you won’t need to change any code outside your model.
Now whenever an instance of your model gets saved, the value stored in the field will get updated with whatever value the decorated function returns.
Example:
class SomeModel(models.Model):
# the other fields
@denormalized(models.CharField,max_length=100)
def some_computation(self):
# your code
return some_value
in this example SomeModel
will have a CharField
named some_computation
.
Note: You must add the column in the DB yourself (either manually or through a south migration) since denorm won’t perform that operation for you.
Adding dependency information¶
The above example will only work correctly if the return value of the decorated function only depends on attributes of the same instance of the same model it belongs to.
If the value somehow depends on information stored in other models, it will get out of sync as those external information changes.
As this is a very undesirable effect, django-denorm-iplweb provides a mechanism to tell it what other model instances will effect the computed value. It provides additional decorators to attach this dependency information to the function before it gets turned into a field.
Denormalizing ForeignKeys¶
If you wish to denormalize a ForeignKey (for example to cache a relationship that is through another model), then your computation should return the primary key of the related model. For example:
class SomeOtherModel(models.Model):
third_model = models.ForeignKey('ThirdModel')
class SomeModel(models.Model):
# the other fields
other = models.ForeignKey('SomeOtherModel')
@denormalized(models.ForeignKey,to='ThirdModel',blank=True, null=True)
@depend_on_related('SomeOtherModel')
def third_model(self):
return self.other.third_model.pk
Callbacks are lazy¶
Your fields won’t get updated immediately after making changes to some data.
Instead potentially affected rows are marked as dirty in a special table and the
update will be done by the denorm.flush
method.
Post-request flushing¶
The easiest way to call denorm.flush
is to simply do it after every completed request.
This can be accomplished by adding DenormMiddleware
to MIDDLEWARE_CLASSES
in your settings.py
:
MIDDLEWARE_CLASSES = (
...
'django.middleware.transaction.TransactionMiddleware',
'denorm.middleware.DenormMiddleware',
...
)
As shown in the example, I recommend to place DenormMiddleware
right after TransactionMiddleware
.
Using the queue¶
If the above solution causes problem like slowing the webserver down because
denorm.flush
takes to much time to complete, you can use a background process to update the data.
The process will check for dirty rows as it is being run and then it will re-check
as soon as it gets a NOTIFY signal from the database server.
To run the process, which waits for NOTIFY command from PostgreSQL server, run:
./manage.py denorm_queue
The command runs in foreground. If you need to daemonize it, use specialized tools like supervisord
It could be tempting to run multiple of such processess - to perform flushing in a multi-threaded manner.
Your objects could depend on other objects, and those objects could depend on even more
objects. As django-denorm-iplweb tries to be a general-purpose tool, at this moment running
multiple instances of denorm_queue
is not recommended. In some situations this could be
perfectly doable - in some, this could easily be a source of database deadlocks. Your milleage
may vary - proceed with caution.
Final steps¶
Now that the models contain all information needed for the denormalization to work, we need to do some final steps to make the database use it. As django-denorm-iplweb uses triggers, those have to be created in the database with:
./manage.py denorm_init
This should be redone after every time you make changes to denormalized fields. On the other hand,
unless you set DENORM_INSTALL_TRIGGERS_AFTER_MIGRATE
variable to False
, trigger installation
will be performed every single time after migrate
command is finished.
Testing denormalized apps¶
When testing a denormalized app you will need to instal the triggers in the setUp method. You could also use a tearDown procedure like:
from denorm import denorms
class TestDenormalisation(TestCase):
def setUp(self):
denorms.install_triggers()
def tearDown(self):
denorms.drop_triggers()