Suppose your domain is “example.com”. In the Google Domains DNS panel select the Google Domain Name Servers, and enter any records that you need to, including an A record for your own server, which we will assume is “ns.example.com”. You will need two “extra” records as well. The listing in Listing 1 may help.
Go ahead and enable DNSSEC. Now you are done with Google Domains, (but don't lose the access credentials)!