Select Mechanical Outline. Under the Properties menu, Change the Mechanical outline layer to the user-defined board outline layer.
To create a custom board shape, select the board outline that has been created. On the Design Tab, in the board shape, select Define from selected Objects.
To create a cutout hole, select the desired outline. On the Tool Tab, in the Convert, select Create Boardcutout from selected primitives.