Struct chumsky::stream::Stream

source ·
pub struct Stream<'a, I, S: Span, Iter: Iterator<Item = (I, S)> + ?Sized = dyn Iterator<Item = (I, S)> + 'a> { /* private fields */ }
Expand description

A type that represents a stream of input tokens. Unlike Iterator, this type supports backtracking and a few other features required by the crate.

Implementations§

source§

impl<'a, I, S: Span, Iter: Iterator<Item = (I, S)>> Stream<'a, I, S, Iter>

source

pub fn from_iter(eoi: S, iter: Iter) -> Self

Create a new stream from an iterator of (Token, Span) pairs. A span representing the end of input must also be provided.

There is no requirement that spans must map exactly to the position of inputs in the stream, but they should be non-overlapping and should appear in a monotonically-increasing order.

source

pub fn fetch_tokens(&mut self) -> impl Iterator<Item = (I, S)> + '_where (I, S): Clone,

Eagerly evaluate the token stream, returning an iterator over the tokens in it (but without modifying the stream’s state so that it can still be used for parsing).

This is most useful when you wish to check the input of a parser during debugging.

source§

impl<'a, I: Clone, S: Span + 'a> Stream<'a, I, S, Box<dyn Iterator<Item = (I, S)> + 'a, Global>>

source

pub fn from_nested<P: 'a, Iter: Iterator<Item = (P, S)>, Many: Iterator<Item = (P, S)>, F: FnMut((P, S)) -> Flat<(I, S), Many> + 'a>( eoi: S, iter: Iter, flatten: F ) -> Self

Create a new Stream from an iterator of nested tokens and a function that flattens them.

It’s not uncommon for compilers to perform delimiter parsing during the lexing stage (Rust does this!). When this is done, the output of the lexing stage is usually a series of nested token trees. This functions allows you to easily flatten such token trees into a linear token stream so that they can be parsed (Chumsky currently only support parsing linear streams of inputs).

For reference, here is syn’s TokenTree type that it uses when parsing Rust syntax.

Examples
type Span = std::ops::Range<usize>;

fn span_at(at: usize) -> Span { at..at + 1 }

#[derive(Clone)]
enum Token {
    Local(String),
    Int(i64),
    Bool(bool),
    Add,
    Sub,
    OpenParen,
    CloseParen,
    OpenBrace,
    CloseBrace,
    // etc.
}

enum Delimiter {
    Paren, // ( ... )
    Brace, // { ... }
}

// The structure of this token tree is very similar to that which Rust uses.
// See: https://docs.rs/syn/0.11.1/syn/enum.TokenTree.html
enum TokenTree {
    Token(Token),
    Tree(Delimiter, Vec<(TokenTree, Span)>),
}

// A function that turns a series of nested token trees into a linear stream that can be used for parsing.
fn flatten_tts(eoi: Span, token_trees: Vec<(TokenTree, Span)>) -> BoxStream<'static, Token, Span> {
    use std::iter::once;
    // Currently, this is quite an explicit process: it will likely become easier in future versions of Chumsky.
    Stream::from_nested(
        eoi,
        token_trees.into_iter(),
        |(tt, span)| match tt {
            // For token trees that contain just a single token, no flattening needs to occur!
            TokenTree::Token(token) => Flat::Single((token, span)),
            // Flatten a parenthesised token tree into an iterator of the inner token trees, surrounded by parenthesis tokens
            TokenTree::Tree(Delimiter::Paren, tree) => Flat::Many(once((TokenTree::Token(Token::OpenParen), span_at(span.start)))
                .chain(tree.into_iter())
                .chain(once((TokenTree::Token(Token::CloseParen), span_at(span.end - 1))))),
            // Flatten a braced token tree into an iterator of the inner token trees, surrounded by brace tokens
            TokenTree::Tree(Delimiter::Brace, tree) => Flat::Many(once((TokenTree::Token(Token::OpenBrace), span_at(span.start)))
                .chain(tree.into_iter())
                .chain(once((TokenTree::Token(Token::CloseBrace), span_at(span.end - 1))))),
        }
    )
}

Trait Implementations§

source§

impl<'a, T: Clone> From<&'a [T]> for Stream<'a, T, Range<usize>, Box<dyn Iterator<Item = (T, Range<usize>)> + 'a>>

source§

fn from(s: &'a [T]) -> Self

Converts to this type from the input type.
source§

impl<'a, T: Clone, const N: usize> From<&'a [T; N]> for Stream<'a, T, Range<usize>, Box<dyn Iterator<Item = (T, Range<usize>)> + 'a>>

source§

fn from(s: &'a [T; N]) -> Self

Converts to this type from the input type.
source§

impl<'a> From<&'a str> for Stream<'a, char, Range<usize>, Box<dyn Iterator<Item = (char, Range<usize>)> + 'a>>

source§

fn from(s: &'a str) -> Self

Please note that Chumsky currently uses character indices and not byte offsets in this impl. This is likely to change in the future. If you wish to use byte offsets, you can do so with Stream::from_iter.

source§

impl<'a, T: Clone + 'a, const N: usize> From<[T; N]> for Stream<'a, T, Range<usize>, Box<dyn Iterator<Item = (T, Range<usize>)> + 'a>>

source§

fn from(s: [T; N]) -> Self

Converts to this type from the input type.
source§

impl<'a> From<String> for Stream<'a, char, Range<usize>, Box<dyn Iterator<Item = (char, Range<usize>)>>>

source§

fn from(s: String) -> Self

Please note that Chumsky currently uses character indices and not byte offsets in this impl. This is likely to change in the future. If you wish to use byte offsets, you can do so with Stream::from_iter.

source§

impl<'a, T: Clone + 'a> From<Vec<T, Global>> for Stream<'a, T, Range<usize>, Box<dyn Iterator<Item = (T, Range<usize>)> + 'a>>

source§

fn from(s: Vec<T>) -> Self

Converts to this type from the input type.

Auto Trait Implementations§

§

impl<'a, I, S, Iter: ?Sized> RefUnwindSafe for Stream<'a, I, S, Iter>where I: RefUnwindSafe, Iter: RefUnwindSafe, S: RefUnwindSafe,

§

impl<'a, I, S, Iter: ?Sized> Send for Stream<'a, I, S, Iter>where I: Send, Iter: Send, S: Send,

§

impl<'a, I, S, Iter: ?Sized> Sync for Stream<'a, I, S, Iter>where I: Sync, Iter: Sync, S: Sync,

§

impl<'a, I, S, Iter: ?Sized> Unpin for Stream<'a, I, S, Iter>where I: Unpin, Iter: Unpin, S: Unpin,

§

impl<'a, I, S, Iter: ?Sized> UnwindSafe for Stream<'a, I, S, Iter>where I: UnwindSafe, Iter: UnwindSafe, S: UnwindSafe,

Blanket Implementations§

source§

impl<T> Any for Twhere T: 'static + ?Sized,

source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
source§

impl<T> Borrow<T> for Twhere T: ?Sized,

const: unstable · source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
source§

impl<T> BorrowMut<T> for Twhere T: ?Sized,

const: unstable · source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
source§

impl<T> Chain<T> for T

source§

fn len(&self) -> usize

The number of items that this chain link consists of.
source§

fn append_to(self, v: &mut Vec<T, Global>)

Append the elements in this link to the chain.
source§

impl<T> From<T> for T

const: unstable · source§

fn from(t: T) -> T

Returns the argument unchanged.

source§

impl<T, U> Into<U> for Twhere U: From<T>,

const: unstable · source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

source§

impl<T, U> TryFrom<U> for Twhere U: Into<T>,

§

type Error = Infallible

The type returned in the event of a conversion error.
const: unstable · source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
source§

impl<T, U> TryInto<U> for Twhere U: TryFrom<T>,

§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
const: unstable · source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.