Text Processor

What this block does

The Text Processor block exposes four UI-selected operations:

• normalize_text
• remove_text
• text_ranker
• text_matching

The selected operation controls both the block configuration and the exact keys allowed in msg.payload.

Inputs and configuration

1. Normalize Text (normalize_text)

Configure in the block UI:

• normalization_method: lemmatize or stemmer

Required input in msg.payload:

• text (string)

No other input keys are accepted for this mode.

2. Remove Text (remove_text)

Configure in the block UI:

• select:
  • remove_html_tag
  • remove_stop_words
  • remove_additional_white_space
  • remove_using_regex
  • remove_trailing_text

Required input in msg.payload:

• text (string)

Required only for remove_using_regex and remove_trailing_text:

• pattern (string)

For the other remove modes, pattern must not be present.

3. Text Ranking (text_ranker)

Configure in the block UI:

• model_id: ranker_deep or ranker_fast
• threshold: optional default threshold from the block editor

Required input in msg.payload:

• query (string)
• rank_items (string[])

Optional input in msg.payload:

• threshold (number from 0 to 1)

If msg.payload.threshold is provided, it overrides the threshold configured in the block.

4. Text Matching (text_matching)

Configure in the block UI:

• operation: similarity or deduplication
• model_id:
  • for similarity: matcher_semantic or matcher_string
  • for deduplication: the handler uses semantic deduplication internally
• threshold

For operation = similarity, required input in msg.payload:

• source_text (string)
• target_text (string)

For operation = deduplication, required input in msg.payload:

• target_texts (string[])

Optional input in msg.payload:

• threshold

Threshold rules:

• Semantic matching uses 0 to 1
• String matching uses 0 to 100

Outputs

The block sends its result directly as msg.payload.

• normalize_text: processed string
• remove_text: processed string
• text_ranker: object with ranked and details
• text_matching with similarity: object with score and passed
• text_matching with deduplication: object with unique_texts

Examples

Normalize text

Input (msg.payload)

{
  "text": "Running tests on better normalized words"
}

Output (msg.payload)

"run test on better normal word"

Remove text with regex

Input (msg.payload)

{
  "text": "Order ID: 99123",
  "pattern": "[0-9]+"
}

Output (msg.payload)

"Order ID: "

Rank candidates

Input (msg.payload)

{
  "query": "refund policy",
  "rank_items": [
    "Our refund policy allows returns within 30 days.",
    "Shipping usually takes 5 business days."
  ],
  "threshold": 0.6
}

Output (msg.payload)

{
  "ranked": [
    "Our refund policy allows returns within 30 days."
  ],
  "details": {
    "Our refund policy allows returns within 30 days.": {
      "rank": 1,
      "score": 0.78
    }
  }
}

Similarity check

Input (msg.payload)

{
  "source_text": "refund period is thirty days",
  "target_text": "returns are allowed within 30 days"
}

Output (msg.payload)

{
  "score": 0.82,
  "passed": true
}

Deduplicate text list

Input (msg.payload)

{
  "target_texts": [
    "invoice total is 120",
    "invoice total is $120",
    "shipping date is tomorrow"
  ]
}

Output (msg.payload)

{
  "unique_texts": [
    "invoice total is 120",
    "shipping date is tomorrow"
  ]
}

Limitations

• Validation is strict: unexpected keys can cause the request to fail.
• text_ranker accepts only query, rank_items, and optional threshold.
• text_matching accepts different input keys depending on whether you chose similarity or deduplication.

Common mistakes

• Sending pattern for remove modes that do not use it.
• Sending extra keys that are not accepted by the selected operation.
• Using a 0-1 threshold for string similarity mode, which expects 0-100.
• Expecting deduplication to use arbitrary matching models; the request handler switches to semantic deduplication internally.