At first glance it looks like those on Thorfinn's side making their own people doubtful of the islanders with a purely intentional purpose, but no it does make sense if those islanders do think along those lines. After all they really are invaders.
Good thing a big chunk of them aren't a bit hostile towards Thorfinn's men and the things they're doing, but you never know as one bad apple spoils the whole barrel, and it was quite evident towards the end too.