From what you have said, namely, that you will not allow any speaker to bring in irrelevant matters, I think there should be no question of any time-limit. If you find after two minutes that a speaker is irrelevant he should be asked to come back to the point or to close.
