Skip to content

NiFi development guide

This guide is for contributors changing NiFi flows, custom processor scripts, or Python extension processors in this repository.

Scope

Use this page for day-to-day development tasks:

  • modifying NiFi templates and controller-service wiring,
  • adding/updating script-based processors (ExecuteStreamCommand),
  • adding/updating Python extension processors (FlowFileTransform),
  • running local quality checks before opening a PR.

Quick start

From the repository root:

cd deploy
source export_env_vars.sh
make start-nifi

Then open NiFi at https://localhost:8082.

Repository map

  • nifi/user_templates/: exported workflow templates.
  • nifi/user_scripts/processors/: script-based processors run by ExecuteStreamCommand.
  • nifi/user_scripts/utils/: shared helpers for scripts/extensions.
  • nifi/user_python_extensions/: Python extension processors exposed in the NiFi UI.
  • nifi/user_scripts/tests/: tests and local helper scripts.
  • docs/nifi/: NiFi-facing developer and usage docs.

Pick the right processor approach

Use script-based processors (ExecuteStreamCommand) when:

  • you need a fast stdin->stdout transform,
  • you do not need first-class processor properties/relationships in the UI.

Use Python extension processors when:

  • you want the processor to appear as a native NiFi processor,
  • you need explicit processor properties and stronger integration.

See also:

Development loop

Script-based processors

  1. Edit/create a script in nifi/user_scripts/processors/.
  2. Validate script input/output locally.
  3. Configure/update ExecuteStreamCommand in NiFi.
  4. Run flow and inspect success/failure relationships.

Local test example:

echo '[{"text":"hello"}]' | python3 nifi/user_scripts/processors/clean_doc.py text_field_name=text

Python extension processors

  1. Edit/create a processor in nifi/user_python_extensions/.
  2. Restart the NiFi service (or reload processors in UI) to pick up changes.
  3. Reconfigure/test the processor in the flow.

If dependencies changed:

  1. Update nifi/requirements.txt.
  2. Rebuild the NiFi image:
bash nifi/recreate_nifi_docker_image.sh

Quality checks

Run the same core checks as CI:

python -m ruff check scripts nifi/user_python_extensions --select F,E9
python -m mypy --config-file pyproject.toml nifi/user_python_extensions scripts

Optional test run:

python -m pytest nifi/user_scripts/tests -q

Debugging

  • NiFi logs:
docker logs -f --tail 300 cogstack-nifi
  • In the NiFi UI, use processor bulletins and data provenance to inspect failures.
  • For script processors, write useful errors to stderr and exit non-zero.

Where to document available processors

Recommended location: docs/nifi/.

  • Keep implementation guidance in:
  • docs/nifi/processor_scripting.md
  • docs/nifi/user_scripts.md
  • docs/nifi/user_python_extensions.md
  • Keep processor inventory on processors catalog.

For each processor entry, include:

  • processor name and source file path,
  • processor type (script-based or extension),
  • expected input and output format,
  • required properties/arguments,
  • common failure modes and example usage.