A non-custodial exchange is an exchange where custody of funds has been decentralised. Another way of describing this is that the service is trustless, i.e. it does not require the involvement of a central authority to secure funds and ensure that trades are executed, cleared and settled correctly. This responsibility is instead given to a blockchain smart contract.
Smart contracts are computer protocols which digitally facilitate, verify, and enforce the negotiation or performance of a contract between two parties. Smart contract logic is limited to a certain number of actions, e.g. the balances updates needed to transfer funds between two users' accounts, and these actions are audited and transparently verifiable as being free from interference by the creator of the logic.
Users self-custody their digital tokens in their blockchain wallet and allocate their funds to a smart contract, which functions as an escrow, when they want to trade them for other digital assets.
During order placement, matching, execution and settlement, users' funds remain locked in the smart contract and never fall under the control of the exchange or another person other than the owners of the funds. In this way non-custodial exchanges, through the capabilities of smart contracts, enable peer-to-peer trading and eliminate exchange and counter-party risk.