Is it Halachicaly acceptable to inform someone in advance that you will charge them for using an item of yours without permission?
I.e. a book, you never charged others to borrow, but because a person is using it without permission, the owner would like to stop such behavior by setting a fine.
One is not allowed to use someone else’s belongings without permission. There are certain circumstances when one may use another’s items without first asking permission (for the assumption is that the owner wouldn’t mind). An example would be, one is allowed to borrow a Talis or Tefilin without the knowledge of the owner so that they can used for a Mitzvah. However, this is only if it is returned to the place and in the way it was originally found. Another example is in a dormitory, borrowing a pen or tissues, for everyone is generally OK with people taking these things.
However, regarding something that is valuable and can get ruined (as in your example a book), one should not use it without permission from the owner. However, if others are seen borrowing it, and it’s known that the owner doesn’t mind, then it would be OK.
If someone generally doesn’t mind people taking (in our case) a book, but decides at some point to stop lending it out, they should put a notice so that others won’t be liable on the Aveirah of Gezel unknowingly.
One is allowed to rent out a book at their own discretion. However, it is not OK to charge a fine for someone who borrowed a book without permission, for this would need to be decided by a proper Beis Din.
ראה גם הליכות בין אדם לחברו ע’ רפב-רפג.