The defendant shall not be liable for such annual value or damages for any period longer than five years before the action or suit was brought, except that, if the sum allowed by the jury for the improvements exceed that allowed to the plaintiff for the annual value and damages of the premises under section three of this article, the jury may then estimate against the defendant such annual value and damages for the time he has used and occupied the same before the said five years.