Friday, April 22, 2011

Grails Envers Plugin

***Update - 4/11/2013***

I finally got around to updating and publishing this plugin: http://www.lucasward.net/2013/04/grails-envers-plugin-update.html



I recently needed to add auditing to my grails application and decided to use envers, mostly for its ability to keep updates to multiple hibernate objects tied together in a way that is query-able afterwards. Unfortunately, because Grails exerts somewhat tight control of Hibernate, and because of how Envers was implemented, getting it to work with grails can be a bit tricky. I did a cursory glance around and couldn't find anyone that had created a plugin to help with Envers and Grails. So while implementing it in my application, I wrote the envers specific code as a plugin, which I have published on github:

https://github.com/lucaslward/grails-envers-plugin

Eventually, I need to write actual documentation for it, and try and get it in the official grails plugin repository. However, for now, I'll write up how to use it based on the integration tests.

If you're pulling it down from the git repo above, you need only do a simple 'grails package-plugin', which will create a zip file, which can be then be installed into your grails application via: 'grails install-plugin /path/to/zip/grails-envers-plugin-0.1.x.zip'. The plugin sets up envers via Hibernate event listeners, and provides gorm style dynamic methods that can be used to query for revisions.

In order to configure your domain objects to use Envers, you need to annotate them with @Audited. (I haven't written any code to try and make that work with static members a la gorm) I'll use the sample domain from the plugin:
@Audited
class Customer {

  String email
  String name
  Address address
  SortedSet orders = new TreeSet()

  static constraints = {
    address (blank: true, nullable: true)
    address component: true
  }

  static mapping = {
    orders cascade: "all,delete-orphan"
  }
}

@Audited
class Address {
  String city
  String zip
}

@Audited
class OrderEntry implements Comparable{

  Date date
  double amount
  int numberOfItems
  Customer customer

  static belongTo = [customer:Customer]

  @Override
  int compareTo(OrderEntry o) {
    date.compareTo(o.date)
  }
}
In this example, I have three domain classes: Customer, Address, and OrderEntry. Customer in this class is the main class, who has a one to one relation ship with an Address, and a collection of Orders. Now, let's assume that we create a customer with an address, then modify the customer and the address twice in separate transactions:

Customer customer
Customer.withTransaction {
  def address = new Address(city: "Chicago", zip: "60640")
  address.save()
  customer = new Customer(name: "PureGorm", email: "tester@gorm.org", address: address)
  customer.save(flush: true)
}

Customer.withTransaction {
  customer = Customer.findByName("PureGorm")
  customer.email = "tester2@gorm.org"
  customer.address.city = "New York"
  customer.save(flush: true)
}

Customer.withTransaction {
  customer = Customer.findByName("PureGorm")
  customer.email = "tester3@gorm.org"
  customer.address.zip = "10003"
  customer.save(flush: true)
}

Note: If you're trying to test envers in your application you have to turn off transactions for your test and wrap individual calls in transactions as above. (with an appropriate tear-down to clean up afterwards of course)

In this scenario, we should have 3 entries in the audit tables for Customer and Address (customer_aud and address_aud respectively). We can query for this methods with the findAllRevisions dynamic method:

def results = Customer.findAllRevisions()

assert results.size() == 3
def r = results[0]
assert r.name == "PureGorm"
assert r.email == "tester@gorm.org"
assert r.address.city == "Chicago"
assert r.address.zip == "60640"
assert r.revisionType == RevisionType.ADD
r = results[1]
assert r.email == "tester2@gorm.org"
assert r.address.city == "New York"
assert r.revisionType == RevisionType.MOD
r = results[2]
assert r.email == "tester3@gorm.org"
assert r.address.zip == "10003"
assert r.revisionType == RevisionType.MOD


As you can see, findAllRevisions returns an array of three results. The plugin will automatically take the envers RevisionEntity and RevisionType, and set them on the returned revisions, which is how revisionType is being checked. As you can see, all three revisions are present. By default its sorted by the earliest revision first. First is an add, followed by two 'mods' (update). The revision entity is also accessible via the revisionEntity property: r.revisionEntity.getRevisionDate(). It's also worth noting that even a custom envers RevisionEntity will be accessible this way. (See the envers documentation for more details on how to do this)

findAllRevisions also supports sorting in the same way as gorm queries. Assuming the same audit trail as above, you could query it this way:

def results = Customer.findAllRevisions(sort:"email",order:"asc")

assert results.size() == 3
assert results[0].email == "tester2@gorm.org"
UserRevisionEntity entity = results[0].revisionEntity
assert entity.getUserId() == currentUser.id
assert results[0].revisionEntity.getUserId() == currentUser.id
assert results[1].email == "tester3@gorm.org"
assert results[2].email == "tester@gorm.org"


You still get back 3 results. But this time the first result is the first update, followed by the second, following by the initial create. This is because its being sorted by email ascending.

As with sorting, you can also use max and offset:

    
def results = Customer.findAllRevisions(max:1)

assert results.size() == 1
assert results[0].email == "tester@gorm.org"

results = Customer.findAllRevisions(max:1,offset:1)

assert results[0].email == "tester2@gorm.org"

results = Customer.findAllRevisions(max:1,offset:2)

assert results[0].email == "tester3@gorm.org"


Besides sorting and paginating, you can also search for revision by property name:

Customer.findAllRevisionsById(customer.id)

Customer.findAllRevisionsByAddress(customer.address)

Customer.findAllRevisionsByName("PureGorm",[sort:"email",order:"asc", max:2])


The first case above is searching by id, and requires a long. The second is searching by the address, which requires an attached hibernate entity. The final is searching by a simple property name, in this case: "PureGorm", with some additional sorting, etc tacked on. As with gorm, you can use any property name on the object to query by. At this point, the plugin only supports querying by one property. If you need to query by more, you will need to use the Envers classes directly. (i.e. AuditReader)

The final method available statically is getCurrentRevision():

Customer.getCurrentRevision()


Which gets the current revision number. Meaning, what is the last revision of any customer?

There are also two methods that are applicable on domain class instances as well:

     
List revisions = customer.getRevisions()
assert revisions != null
assert revisions.size() == 3

Customer oldCustomer = customer.findAtRevision(revisions[1])
assert oldCustomer.email == "tester2@gorm.org"
assert oldCustomer.address.city == "New York"


The first method will return all revisions of a particular class. (The same as Customer.findAllRevisionById(customer.id)) And the second will find a particular customer at a particular revision.

There is certainly some functionality I'm missing here, as I'm using the integration tests as an example. I will add more blog entries on the subject as I discover the corner cases I have missed. Hopefully, I can also get time to write actual documentation, and get the plugin in the official repository as well.

30 comments:

DM said...

G'day Lucas
Using eclipse (STS) and running through the blog, after installing the plugin, and attempting to implement the Customer class, the @Audited annotation is marked as an error ... is there some extra configuration I need to do?

DM said...

Okay - sorted. Just need to add:
"import org.hibernate.envers.Audited".

daniel said...

Hi Lucas,
I was recently integrating envers myself into Grails, but had a problem with empty _aud tables. This way I stumbled upon your plugin. Trying it with your plugin unfortunately yields the same result (empty _aud tables). I upgraded your plugin to Grails version 1.3.7 though, but I don't think this is causing the problem.

Any hint for me what is going wrong?

Cheers,
Daniel

Lucas Ward said...

@DM Sorry about that, I guess I left out the imports when I cut and pasted.

Lucas Ward said...

@daniel

I'm assuming by empty _aud tables you mean that when doing an update to a database table, you aren't seeing a corresponding entry in the corresponding audit table?

My first thought is that it has to be something to do with transactions. For example, I have seen this happen with MySql myisam tables. You can write to them without transactions, and since Envers is tied completely to transactional hooks, it will never be called. I'm will to be if you breakpointed the Envers AuditEventListener class, which is what is actually listening to Hibernate events. If you replicate the tests I have in the blog post, using .withTransaction around your calls, you should see the entries. If that works in an integration test, then you need to probably take a look at your Grails controllers. Remember, by default they are not transactional.

daniel said...

Lucas,
that was the problem. When using a grails service, everything works as expected.

Thanks a lot for your help and the plugin ;-)

Daniel

Fábio Miranda said...

Hi Lucas!

Nice post and nice plugin!

The only thing that is not clear is how to configure UserRevisionEntity to track the user that performed changes to the database.

First, only 'rev' and 'revtstmp' columns were created at the database.

Second, even after I found SpringSecurityRevisionListener, SpringSecurityServiceHolder and StubSpringSecurityService in the sources, it's not clear how to use them.

Third, is it necessary to integrate with a particular technology (Spring Security)?

Thanks!
Fábio Miranda.

Lucas Ward said...

@Fabio

Sorry for the delayed response.

Regarding the UserRevisionEntity. It's really tricky. The weirdest thing is it will only work if you make it a java class and register it directly with hibernate. If you look at the plugin code, you'll see a hibernate config file,where the revision entity is the only configured class. I have no idea why @Audit works fine on gorm classes, but @RevisionEntity doesn't. The listener via the annotation is also really silly of them. It prevents the listener from wiring it up. Once the plugin is released completely, and seems to be stable, my next big release will likely be to write my own configuration for it, probably using a combination of normal plugin config in config.groovy and static methods a la gorm on domain classes. But I probably won't even try that until grails 1.4, since envers changes to a first class citizen in hibernate 3.5. (1.4 will go to 3.6 I believe)

Oh, and it isn't necessary to use Spring Security. That's just what I use for my app at work, but really whatever you're using for authentication will work.

morungos said...

This is almost ideal, but I need to be selective about which fields to track for audit. I tried the plugin but it always seemed to cough on a relation to a non-audited object. is this known to work?

Lucas Ward said...

@morungus

The plugin doesn't actually address configuration, and relies on envers itself. As I mentioned in a previous comment, I would love to do more gorm style mappings, but will likely wait until grails 1.4.

This problem is kind of tricky for groovy in general (and why I suspect grails uses static fields). There's a few posts you can find on the topic, but because groovy generates your getters and setters, etc, annotations on a field don't do what you expect (or in this case what envers expects). Your best bet is to create the getters and setters yourself, and annotate them. It's still tricky, but that's your best bet for individual field annotations on gorm classes. Of course, it will work straight away on java classes configured with hibernate xml or entity annotations. Although, I'm sure that's not a great option for many. You can test this yourself by annotating your groovy classes and using reflection in java to see if you can get at them.

morungos said...

Thanks Lucas. I'd be OK doing that if I was a little more Groovy-savvy. Couldn't find much in the way of annotation at anything other than the class level. I was about 2/3 way through improvising auditing with Groovy MOP hacks, an event listener, and a static field, so I might for now go that way, but I'll certainly track this plugin, which appears to be much better for the long term.

euvitudo said...

Regarding the transactional context:

I've been experimenting with your plugin (thanks for the plugin, btw), and have only simplistic examples using grails scaffolding. I found that after adding @Transactional (org.springframework.transaction.annotation.Transactional) to the controllers, updates to the audit tables are committed.

The drawback to this method (at least, while using scaffolding) is that all methods will be transactional.

Annotating a service or service method in this way will do the same.

Lucas Ward said...

@euvitudo

Transactions in Grails are a bit weird anyway, but yes, nothing will be written to the audit tables until a commit. That's because a common revision ties together all the work done in a single transaction. Of course, if you do this work in a service, that method is transactional, you can use the @transactional annotation as you already mentioned, and there is a transactional controller plugin that let's you set which methods on a controller should be transactional. (which is what I use in my app)

Smita Saraswat said...

Lucas,

I am running grails 2.0M2 version and when I installed your plugin, I get the following error :

Caused by ClassCastException: org.codehaus.groovy.grails.orm.hibernate.cfg.GrailsAnnotationConfiguration cannot be cast to org.hibernate.cfg.AnnotationConfiguration

Thanks.

Smita Saraswat said...

Anyone as run this plug in with the latest version of grails ?
Thanks.

gmacmullin said...

I've been using the Envers plugin with Graisl 1.3.7, but I have to download the source code and upgrade it to Grails 1.3.7

The weird thing is the plugin works perfectly when I run "grails run-app", but when I deploy it as a war file in Tomcat, the _aud tables are not created. I turned on logging for Envers and the plugin, but I don't see any errors/problems.

Lucas, have you had any problems running the plugin in a WAR file?

Thanks,
Glen

gmacmullin said...

Looks like I had a typo where I declared the Envers plugin as a compile plugin dependency. It works great now.

Thanks for creating the plugin Lucas!

Glen

Lucas Ward said...

Sorry for the delay in responding to some of these. I took a long vacation and have just now gotten back.

@Smita - The plugin definitely won't work below grails 1.3, and even that can get tricky at times. The problem is that Envers was integrated with hibernate a couple of years ago. Grails is really behind on hibernate releases (for obvious reasons). It will all be much more straightforward with Grails 2.0 I think.

@Gmacmullin, I'm glad it worked out for you. I really need to get this thing into the official plugin repo...

korsbecker said...

Smita,

Did you solve your problem with

Caused by ClassCastException: org.codehaus.groovy.grails.orm.hibernate.cfg.GrailsAnnotationConfiguration cannot be cast to org.hibernate.cfg.AnnotationConfiguration

I have exact the same problem with Grails 2.0. I have upgraded the plugin to to grails 2.0 and change to Hibernate 3.6.7.Final and explicitly added a runtime for hibernate-envers:3.6.7.Final

But still with the same problem??

cheers
henrik

Lucas Ward said...

korsbecker,

Getting the plugin to work with Grails 2.0 is a whole other problem. Grails 2 jumped the hibernate version up to 3.6 which is something like 4 years worth of Hibernate. I haven't looked into that at all yet (but probably will in 3 or 4 months) but there's no telling what issues might pop up. The error you have looks more related to issues cause by this upgrade than from anything else.

Manuel B. said...

Jay Hogan has forked the repository and upgrade envers plugin for grails 2.0

URL: https://github.com/jayhogan/grails-envers-plugin

Regards

Tomasz Kalkosiński said...

Hi Lucas

I wrote an interesting blog post about working with Grails 2.1 and Hibernate Envers together. It'd be great if you can read, comment and spread a word about my work. My post can be found here:

http://refaktor.blogspot.com/2012/08/hibernate-envers-with-grails-210.html

Greetings,
Tomasz Kalkosiński

Prashant Potluri said...

If there are multiple data sources defined in DataSource.groovy, the plugin is not saving any data in the audit tables. Any suggestions?

Lucas Ward said...

Usually when it's not pushing to the AUD tables it's a transactional issue, because it only pushes those entries on transactional boundaries.

Prashant Potluri said...

There is @Transactional on controller. Also tried the programmatic way using Entity.withTransaction.
I still don't see any data written to audit tables. DataSource.groovy contains multiple data sources:

development {
dataSource {
dbCreate = "update"
url = "jdbc:mysql://localhost/dev"
username = "xxxxx"
password = "xxxx"
}
dataSource_audit {
dbCreate = "update"
url = "jdbc:mysql://localhost/audit"
username = "xxxx"
password = "xxxx"
}
}

Prashant Potluri said...

I am using @Audited on getter methods instead of at class level for auditing specific properties only. However i am unable to use any of the dynamic finder methods findAllRevisionsById() etc. It says cannot resolve property revision. If i annotate the class @Audited, it works.
Is there a work around for this without annotating the class?

bdbull said...

Has anyone come up with an efficient way to add a "revision" property to the domain class? It seems a little involved if I want to actually attach the revision to the domain class itself.

jolo said...

Hi Lucas,

if I want to use "findAllRevisions()" or any of the "findAll.." queries, I get an error:

Cannot query [Hotel] on non-existent property: revisions

The "Hotel" domain class has the @Audited annotation already, and the save() is done by

Hotel.withTransaction{}

closure.

Any hint how to solve or work around this issue?

Yours Johannes

Lucas Ward said...

Jolo, can you provide more details, like what version of grails you're using, what version of the plugin, etc?

Also, is this the only class having the issue?

jolo said...

Hi Lucas,

I'am using grails 2.2.4 and the envers 2.1.0 plugin.
The issue is generally with every domain class.
Meanwhile I suspect, that it might be related with the injection of sessionFactory, but I am not quite sure about that.

Yours Johannes