Passport 'safekeeping'

So my company when I joined a year back kept the passport with them, and now I decided to go to my hometown for 5 days and before giving me my passport they took away my company lD which is used to like log in (although on current site where I work I don't need it) and made me sign a piece of paper which says I agree to handover the passport to the company for 'safekeeping'. Ik the company can't keep the passport as its illegal but since I signed this shitty paper can that be used against me?