Your organization must inventory and track all physical access devices. We suggested in PS.L2-3.9.2 to start an inventory for each employee of mobile devices and system access, and assignment of keys and other physical access devices must be part of this list which can then be reviewed to ensure collection of all items when the employee departs the company.
You should also train all employees to report any lost or stolen keys, badges, etc., as soon as it is noticed they are missing.
Your policy will note the requirement for this inventory, and your plan for level three will note who is responsible for the inventory, and where the inventory is kept. You should conduct audit/inventory of all physical access devices (keys, badges) on a regular basis to measure your policy and plan's effectiveness.
Do not forget to regularly review the electronic access systems themselves to ensure that only the appropriate people still have access to controlled areas.