## NAME

ACE_Tokenizer - Tokenizer

## SYNOPSIS

    #include <ace/SString.h>

    class ACE_Tokenizer
    {
      public:
        ACE_Tokenizer (LPTSTR buffer);
        int delimiter (TCHAR d);
        int delimiter_replace (TCHAR d, TCHAR replacement);
        int preserve_designators (TCHAR start, TCHAR stop, int strip=1);
        LPTSTR next (void);
        MAX_PRESERVES=16 }; protected: int is_delimiter (
            TCHAR d,
            int &replace,
            TCHAR &r
            );
        int is_preserve_designator (TCHAR start, TCHAR &stop, int &strip);

        class Delimiter_Entry
        {
          public:
            TCHAR delimiter_;
            TCHAR replacement_;
            int replace_;
        };

`private:` `LPTSTR buffer_;` `int index_;`

class Preserve_Entry { `public:` `TCHAR start_;` `TCHAR stop_;` `int strip_;` `};` `Preserve_Entry preserves_[MAX_PRESERVES];` `int preserves_index_;` `Delimiter_Entry delimiters_[MAX_DELIMITERS];` `int delimiter_index_;` `};`

## DESCRIPTION

Tokenizes a buffer. Allows application to set delimiters and preserve designators. Does not allow special characters, yet (e.g., printf ("\\"like a quoted string\\"")).

## PUBLIC MEMBERS

    ACE_Tokenizer (LPTSTR buffer);

`buffer` will be parsed.

    int delimiter (TCHAR d);

`d` is a delimiter. Returns 0 on success, -1 if there is no memory left.

    int delimiter_replace (TCHAR d, TCHAR replacement);

`d` is a delimiter and, when found, will be replaced by `replacement`. Returns 0 on success, -1 if there is no memory left.

    int preserve_designators (TCHAR start, TCHAR stop, int strip=1);

For instance, quotes, or '(' and ')'. Returns 0 on success, -1 if there is no memory left. If `strip` == 1, then the preserve designators will be stripped from the tokens returned by next.

    LPTSTR next (void);

Returns the next token.

    MAX_PRESERVES=16 }; protected: int is_delimiter (
        TCHAR d,
        int &replace,
        TCHAR &r
        );

Returns 1 if `d` is a delimiter, 0 otherwise. If `d` should be replaced with `r`, `replace` is set to 1, otherwise 0.

    int is_preserve_designator (TCHAR start, TCHAR &stop, int &strip);

If `start` is a start preserve designator, returns 1 and sets `stop` to the stop designator. Returns 0 if `start` is not a preserve designator.

## PRIVATE MEMBERS

    LPTSTR buffer_;

    int index_;

    Preserve_Entry preserves_[MAX_PRESERVES];

The application can specify MAX_PRESERVES preserve designators.

    int preserves_index_;

Pointer to the next free spot in preserves_.

    Delimiter_Entry delimiters_[MAX_DELIMITERS];

The tokenizer allows MAX_DELIMITERS number of delimiters.

    int delimiter_index_;

Pointer to the next free space in delimiters_.

## AUTHOR

Douglas C. Schmidt (schmidt@cs.wustl.edu)

## LIBRARY

ace