Can I turn off Caller ID so it doesn't show up in the app when I get an incoming call?

The Caller ID is displayed only if the PBX sends it in the invite (in the "From" header) or if a contact with this phone number exists on the Ringotel platform (or connected integration). If there is no contact with this phone number, you can try configuring it on the PBX so that it does not send the Caller ID in the “From” header.