That is entirely something あなた should be discussing with your Significant Other, and depends on the terms of the relationship. によって all means, あなた do you, and don't let anyone dictate who あなた are フレンズ with, but do keep in mind the feelings of your S.O. and be courteous and mature about your habits. If s/he is uncomfortable, allow them to express that and don't get defensive about it. Just listen. If s/he has no problem with it, then at least あなた were up front about it and now there should be no confusion.