I'm implementing a scanner in Rust. I have a scan
method on a Scanner
struct which takes a string slice as the source code, breaks that string into a Vec<&str>
of UTF-8 characters (using the crate unicode_segmentation
), and then delegates each char to a scan_token
method which determines its lexical token and returns it.
extern crate unicode_segmentation;
use unicode_segmentation::UnicodeSegmentation;
struct Scanner {
start: usize,
current: usize,
}
#[derive(Debug)]
struct Token<'src> {
lexeme: &'src [&'src str],
}
impl Scanner {
pub fn scan<'src>(&mut self, source: &'src str) -> Vec<Token<'src>> {
let mut offset = 0;
let mut tokens = Vec::new();
// break up the code into UTF8 graphemes
let chars: Vec<&str> = source.graphemes(true).collect();
while let Some(_) = chars.get(offset) {
// determine which token this grapheme represents
let token = self.scan_token(&chars);
// push it to the tokens array
tokens.push(token);
offset += 1;
}
tokens
}
pub fn scan_token<'src>(&mut self, chars: &'src [&'src str]) -> Token<'src> {
// get this lexeme as some slice of the slice of chars
let lexeme = &chars[self.start..self.current];
let token = Token { lexeme };
token
}
}
fn main() {
let mut scanner = Scanner {
start: 0,
current: 0,
};
let tokens = scanner.scan("abcd");
println!("{:?}", tokens);
}
The error I receive is:
error[E0597]: `chars` does not live long enough
--> src/main.rs:22:42
|
22 | let token = self.scan_token(&chars);
| ^^^^^ borrowed value does not live long enough
...
28 | }
| - borrowed value only lives until here
|
note: borrowed value must be valid for the lifetime 'src as defined on the method body at 15:17...
--> src/main.rs:15:17
|
15 | pub fn scan<'src>(&mut self, source: &'src str) -> Vec<Token<'src>> {
| ^^^^
I suppose I understand the logic behind why this doesn't work: the error makes it clear that chars
needs to live as long as lifetime 'src
, because tokens
contains slice references into the data inside chars
.
What I don't understand is, since chars
is just a slice of references into an object which does have a lifetime of 'src
(namely source
), why can't tokens
reference this data after chars
has been dropped? I'm fairly new to low-level programming and I suppose my intuition regarding references + lifetimes might be somewhat broken.