Not disabling but making a people/group field picker read only

Dear All,

I want the ability to make my multiple and a single people/group field picker read only.
I have other validations on my form so it cannot be made read only on the side panel of the Designer.
I have different form sets and want some form sets (and their users) not to be able to edit with the people/group field picker but that it is read only when in a certain state.

Can you please help.
I have tried the disable JS option but they can still edit (ticking the x option which removes the user) I don't want them to do that.

Kind regards,

Dear @Dina_Louw,
That shouldn't be the case with a disabled field, editing it or deleting users should be impossible:

Are there any errors in browser's console? What's the version of the app package in the app catalog?