Official answer: No. RTS cannot be mixed with Custom shipping calculations.
Unofficial short answer: Maybe (in theory)
Unofficial long answer:
(DISCLAIMER: This is pure theory based off some experimentation. Please keep that in mind, and test this in a sandbox environment before trying it live.)
My suggestion is that you take your current store, clone it and make sure everything all maps properly - you want it to be identical to your original store, and to stay identical to it even as your catalog / stock / etc. changes. Think about it as running two stores in parallel, or mirroring.
One store will be restricted to the US only via Configuration > Taxes > Edit Countries. You want to ensure only the US (or Canada and US, if you're including Canada in domestic shipping) is selected for use. You also want to disable Guest Checkout via Configuration > Settings > Guest Checkout. This should ensure that people are required to sign up, and can only sign up with addresses from the US.
From there, go into Configuration > Set Up Your Store > Shipping Calculations. Set up your Custom shipping calculations for shipping based on speed as you currently do. Set up your shipping methods.
On the other store, you'll go into Configuration > Taxes > Edit Countries, and select all the companies to which you plan to ship, EXCEPT the US (or US and Canada; you get the idea). Again, you want to disable Guest Checkout. You also may want to change the default at Configuration > Settings > Address.Country.Default.
From there in Configuration > Settings > RtShipping.* you'll go through and configure all the applicable Real Time Shipping settings basedon your shipping methods. Once you have done that, go into Configuration > Set Up Your Store > Shipping Calculations. Select Real Time Shipping, Save, then take a look at your shipping methods. Configure accordingly.
I suspect that to prevent conflict, you may need to run at least one of these from a subdomain, possibly with IP sniffing to determine which way to route a visitor. There might be other ways of doing it, though. You may also need to work with the Localization settings for the one that is to serve foreign visitors.
Some reference material:
Regarding Localization stuff: