Interesting question. I'd like to know the proper way to do this, but this (very) hacky way works for me.
---
output:
html_document:
css: ~/knitr.css
---
```{r, include=FALSE}
library(knitr)
knit_hooks$set(plot = function(x, options) {
fig_fn = paste0(opts_knit$get('base.url'), paste(x, collapse = '.'))
fig.cap <<- knitr:::.img.cap(options)
sprintf("<figure><img src='%s'><figcaption>%s</figcaption></figure>",
fig_fn, fig.cap)
})
```
```{r, fig.cap = 'Figure I: the plot of my figure.'}
plot(1:5)
````
I say some things and some other things.
Oh, yeah please refer to `r fig.cap`

This works for the most recent figure generated, but you could work in a figure counter or something else to make unique variables for each caption so that you can reference whenever you want.