What I want to do is move the background horizontally and have it repeat infinitely.
I tried using an ImageSwitcher with animation to give this effect,          
        
you can use AndroidScrollingImageView Library, all you have to do is to define speed and the drawable source
<com.q42.android.scrollingimageview.ScrollingImageView
    android:id="@+id/scrolling_background"
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    scrolling_image_view:speed="1dp"
    scrolling_image_view:src="@drawable/scrolling_background" />
EDIT:
as @Cliff Burton mentioned, you can rotate the view by 90 deg if you want to scroll vertically.
Why don't you try to just animate the background yourself instead of using a ViewSwitcher? All you need is one simple ValueAnimator:
First add two identical ImageViews to your layout and set the same background image to both of them:
<?xml version="1.0" encoding="utf-8"?>
<FrameLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent">
    <ImageView
        android:id="@+id/background_one"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:src="@drawable/background"/>
    <ImageView
        android:id="@+id/background_two"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:src="@drawable/background"/>
</FrameLayout>
Then use a ValueAnimator to animate their translationX property, but offset them by their width:
final ImageView backgroundOne = (ImageView) findViewById(R.id.background_one);
final ImageView backgroundTwo = (ImageView) findViewById(R.id.background_two);
final ValueAnimator animator = ValueAnimator.ofFloat(0.0f, 1.0f);
animator.setRepeatCount(ValueAnimator.INFINITE);
animator.setInterpolator(new LinearInterpolator());
animator.setDuration(10000L);
animator.addUpdateListener(new ValueAnimator.AnimatorUpdateListener() {
    @Override
    public void onAnimationUpdate(ValueAnimator animation) {
        final float progress = (float) animation.getAnimatedValue();
        final float width = backgroundOne.getWidth();
        final float translationX = width * progress;
        backgroundOne.setTranslationX(translationX);
        backgroundTwo.setTranslationX(translationX - width);
    }
});
animator.start();
This results in an continuous animation which repeats the background indefinitely and should look something like this:
