Strange test isolation problems finally solved!

I have to tell you a boring story about test isolation. On my flight back home from the Cathedral Sprint, I just solved an issue which stole half of the last sprint day's afternoon for Philip Bauer and me. Maybe it prevents someone else of having the same trouble.

Foto by Johannes Raggam (Side street in Mombasa, Kenya)

plone.app.event's tests from the 1.1.x branch were failing for Plone 4.3 but not Plone 5 for absolutely no obvious reason. They were running for 4.3 some time before! We had some isolation problems - content created by test setup methods was still available for after test teardown, so succeeding test methods failed when trying to create these contents again.  That problem persisted, even after Philip and I were changing all tests to use a functional test layer for better isolation.

After long investigations without success, finally a exclusion method helped me. I copied the source directory from the 1.0.x branch into the 1.1.x branch, so that I could easily restore individual files to their newer version with `$ git checkout`. I did that from top-down for the files `$ git status` listed up, running the whole test suite after each checkout.

Luckily I hadn't go down very far until until I found the guilty test.  It was a test which was apparently copied over from somewhere else (I guess CMFPlone) which used ancient test frameworks - Products.PloneTestCase, Testing.ZopeTestCase and an installation of a Product I never heard about before: "Transience".  See this commit for what has changed.

So, if you encounter strange test behavior which you cannot explain, look at your test setup and if some of your tests use old testing frameworks. Maybe this helps...