I think the ScrollViewer assumes that the origin will be at (0, 0) -- after all, there's no way to tell it otherwise; Canvas is the only panel that knows about specified-and-nonzero origins.
A simple option might be to forget about a ScrollViewer, and implement your own panning functionality -- perhaps with dragging, or perhaps with a big "scroll left" button on the left side of the view, a "scroll right" button on the right, etc. -- and implement it in terms of a transform on the content.
But if you need to stick with the classic scrollbar metaphor, I think you could make your own panel (or override ArrangeOverride too, which amounts to the same thing) and offset all the positions -- make them zero-based. So if you have an element at (20, -20) and another at (0, 5), you would need to offset everything downward by 20 to make it fit in a zero-based space; and when you lay out your children, the first would go at (20, 0) and the second at (0, 25).
But now scrolling is weird. If the user is scrolled all the way to the bottom, and drags something off the bottom edge, the view stays put; same with the right. But if they're scrolled all the way to the top, and drag something off the top edge, suddenly everything jumps downward, because the ScrollViewer's VerticalOffset was zero before and is still zero, but zero means something different because of your layout offset.
You might be able to get around the scrolling weirdness by binding the HorizontalOffset and VerticalOffset. If your ViewModel automatically fixed up those properties (and fired PropertyChanged) whenever a control moves "negative", then you might be able to keep the view scrolled to the same logical content, even though you're now telling WPF's layout engine that everything is somewhere else. I.e., your ViewModel would track the logical scroll position (zero, before you drag the element negative), and would report the zero-based scroll position to the ScrollViewer (so after the drag, you would tell the ScrollViewer that its VerticalOffset is now 20).