The validators used in the form fields are served from the modules config:
/config/modules/form/config/validators

So all validators are served for all form templates throughout the system/all sites.

It would be better, if it would be definable on the concrete component itself, which validators should be choosable.
Otherwise its not possible, to provide validators just for a specific site/form.

Most flexible it would be, if they would be definable on the the main form components ('form') which are generally allowed.
And probably on the concrete form-field component constraining of the general defined ones which sot server (if nothing defined, all forms the 'form' paragraph are allowed).