Only Manager of a department and Administrator of the DMS can remove people from departments. So all the users whom you will make manager of the department can remove any user from department. However, every user can remove own self from the department.