I was running into this exact issue trying to get a server upgraded from 9.3 to 9.6. Through a bit of trial and error I found a successful path. Just a detail about the upgrade process: after each upgrade, the RDS requires a manual reboot. I would think the upgrade would automatically handle that, but it does not.
Starting point:
SELECT version(); -> PostgreSQL 9.3.20 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit
SELECT postgis_full_version(); -> POSTGIS="2.1.8 r13780" GEOS="3.5.0-CAPI-1.9.0 r4084" PROJ="Rel. 4.9.2, 08 September 2015" GDAL="GDAL 1.11.5, released 2016/07/01" LIBXML="2.9.1" LIBJSON="UNKNOWN" RASTER
ALTER EXTENSION postgis UPDATE; -> Returned success, but no change to PostGIS version
After upgrading to 9.4:
SELECT version(); -> PostgreSQL 9.4.18 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit
SELECT postgis_full_version(); -> POSTGIS="2.1.8 r13780" GEOS="3.5.0-CAPI-1.9.0 r4084" PROJ="Rel. 4.9.2, 08 September 2015" GDAL="GDAL 1.11.5, released 2016/07/01" LIBXML="2.9.1" LIBJSON="UNKNOWN" RASTER
ALTER EXTENSION postgis UPDATE; -> Returned success, but no change to PostGIS version
After upgrading to 9.5:
SELECT version(); -> PostgreSQL 9.5.13 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit
SELECT postgis_full_version(); -> POSTGIS="2.1.8 r13780" GEOS="3.5.1-CAPI-1.9.1 r4246" PROJ="Rel. 4.9.2, 08 September 2015" GDAL="GDAL 1.11.5, released 2016/07/01" LIBXML="2.9.1" LIBJSON="UNKNOWN" RASTER
ALTER EXTENSION postgis UPDATE; -> SQL Error [XX000]: ERROR: attempt to redefine parameter "postgis.backend"
So I was stuck here for a bit, couldn't seem to get PostGIS upgraded successfully. Then I saw this post describing that same error and decided to just wait for 24 hours to see if the RDS instance received a patch. It seems that it did, though I didn't notice anything in the logs. In any event, without any changes from myself, the next day I was able to successfully upgrade the PostGIS extension
ALTER EXTENSION postgis UPDATE; -> Returned success
SELECT postgis_full_version(); -> POSTGIS="2.2.5 r15298" GEOS="3.5.1-CAPI-1.9.1 r4246" PROJ="Rel. 4.9.3, 15 August 2016" GDAL="GDAL 2.0.3, released 2016/07/01" LIBXML="2.9.1" LIBJSON="0.12" RASTER
After that, I was able to upgrade the RDS instance to 9.6
SELECT version(); -> PostgreSQL 9.6.9 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit
SELECT postgis_full_version(); -> POSTGIS="2.2.5 r15298" GEOS="3.5.1-CAPI-1.9.1 r4246" PROJ="Rel. 4.9.3, 15 August 2016" GDAL="GDAL 2.0.3, released 2016/07/01" LIBXML="2.9.1" LIBJSON="0.12" RASTER
ALTER EXTENSION postgis UPDATE; -> Returned success
SELECT postgis_full_version(); -> POSTGIS="2.3.7 r16523" PGSQL="96" GEOS="3.6.2-CAPI-1.10.2 4d2925d6" PROJ="Rel. 4.9.2, 08 September 2015" GDAL="GDAL 2.1.4, released 2017/06/23" LIBXML="2.9.1" LIBJSON="0.12" RASTER
I'd like to figure out how to trigger that apparent patch to the extensions immediately rather than just waiting for it, but I haven't been able to figure that out yet.