Yes, the first method is correct, the second isn't. Imagine a wall that is perpendicular to your view direction (so you're looking at the side of the wall). Suppose the wall's front is facing to your right. Then, if you move right, the front of the wall becomes visible. However, if you move to the left, the back of the wall becomes visible. And even though the visible features of the wall change as you move, both your and the wall's view direction stay the same. Therefore, using merely those two variables to determine whether to cull the wall must be incorrect.
In fact, you're not even interested in your own view direction. What matters is whether your location with respect to the wall is in front or behind it's plane. Even if you rotate, your position with respect to the wall's plane stays the same. So no matter your view direction, if the wall was facing you, it will continue to face you.