AI solution for JavaScript Coding Question on BFE.dev
119. create a tokenizer

One possible solution to this question is as follows:

function* tokenize(expr) {
  let buffer = '', i = 0
  const isDigit = (ch) => /\d/.test(ch)
  const isOp = (ch) => '+-*/'.indexOf(ch) !== -1

  while (i < expr.length) {
    let ch = expr.charAt(i)
    if (ch === ' ') {
      i++
    } else if (isDigit(ch)) {
      while (isDigit(ch)) {
        buffer += ch
        i++
        ch = expr.charAt(i)
      }
      yield buffer
      buffer = ''
    } else if (isOp(ch)) {
      yield ch
      i++
    } else if (ch === '(' || ch === ')') {
      yield ch
      i++
    } else {
      console.log('Error: unrecognized character')
      yield undefined
      break
    }
  }
  if (buffer !== '') {
    yield buffer
  }
  yield undefined
}

This implementation uses a generator function to produce the tokens of the input expression. It reads the input character by character, and maintains a buffer for digits that are part of a number.

The isDigit() and isOp() helper functions check whether a given character is a digit or an operator, respectively. The main loop of the function reads from the input string and decides what to do depending on the type of character encountered:

• If the character is a space, it is ignored.

• If the character is a digit, we enter a loop that keeps appending digits to the buffer until a non-digit is encountered. Then the buffer is yielded and reset.

• If the character is an operator (+, -, *, /), it is yielded as a token.

• If the character is a parenthesis, it is also yielded as a token.

• If the character is anything else, an error message is printed and the function yields undefined to signal the end of the token sequence.

After the main loop terminates, the function checks whether there are any remaining digits in the buffer, and yields them if necessary. Finally, the function yields undefined again to signal the end of the token sequence.