Why do some Workers leave buildings?

Possible reasons:

1. Workers left without supervision for a long time begin playing truant from work. If a ruler doesn't appear in the Game for 4 days (no matter if there is a Riot) 80% of workers leave their working places.
When the ruler appears the Counsellor will suggest driving idlers to work with one click.

2. After the Gate closing, Workers and Builders leave buildings in villages.

All answers to the FAQ