Log Delivery and Storage

Overview

Zylon supports internal log storage and external log delivery through syslog and HTTP.

Syslog delivery is intended for SIEM applications and log collectors that accept syslog or CEF events.
HTTP delivery is intended for structured ingestion endpoints, cloud log destinations, and customer-managed collection pipelines.
Delivery is asynchronous.
With failurePolicy: "drop", delivery is designed to stay off the critical request path when queues are full.
Delivered events can be filtered by action patterns and organization IDs.
Stored logs stay in the customer’s Zylon deployment unless external log delivery through syslog or HTTP is enabled.

Configure these settings through the Zylon Helm values YAML. The chart maps values to backend runtime settings internally; customer configuration should use the YAML keys shown here. For governance, auditability, EU AI Act traceability support, evidence preservation, and retention guidance, see Auditability & AI Governance: Logging, Evidence, and SIEM Delivery. For manual log retrieval, see Backoffice API: Logging.

At a glance

Channel	Enable with	Format options	Best for
Internal storage	`logging.storage.enabled: true`	Stored structured Zylon log records	Backoffice review and manual API retrieval
Syslog	`logging.delivery.syslog.enabled: true`	`rfc5424_json`, `cef`	SIEM applications, syslog collectors, and CEF collectors
HTTP	`logging.delivery.http.enabled: true`	`canonical_json`	HTTP collectors, ingestion gateways, and cloud log destinations

Syslog and HTTP delivery can be enabled at the same time. When both are enabled, Zylon dispatches matching logs to both destinations. Filters, failure policy, queue, and batch settings are configured separately for each delivery method but follow the same behavior.

Before you configure delivery

Confirm the collector endpoint is reachable from the Zylon backend.
Confirm host or URL and port.
Select transport and format.
Decide TLS certificate verification behavior.
Configure collector-side input, index, parser, or ingestion route.
Understand retention requirements in both Zylon and downstream systems.
Review filters so required audit events are not excluded.

Choose a delivery channel

Requirement	Recommended channel
Existing SIEM or syslog collector input	Syslog
Collector accepts RFC 5424 style syslog messages	Syslog with `rfc5424_json`
Collector accepts CEF events	Syslog with `cef`
Custom HTTP ingestion gateway	HTTP with `canonical_json`
Vendor-specific pipeline transformation	HTTP with downstream mapping

Storage and retention

Internal log storage is controlled under logging.storage.

Storage keys

Key	Default	Values	Description
`enabled`	`true`	`true`, `false`	Enables internal structured log persistence.

Retention keys

Internal log cleanup is controlled by the cleanup scheduler under zylonBackend.scheduler.cleanup. Scheduler settings:

Key	Default	Values	Description
`enabled`	`false`	`true`, `false`	Enables the cleanup scheduler.
`cron`	`0 3 * * *`	cron expression	Cleanup schedule. Shared with hard delete cleanup.
`timezone`	`UTC`	time zone	Time zone used by the cleanup schedule. Shared with hard delete cleanup. If the runtime setting is omitted outside Helm, the backend uses the server JVM time zone.

Under logging:

Key	Default	Values	Description
`enabled`	`false`	`true`, `false`	Enables stored log cleanup.
`deleteAfterDays`	`180`	integer days	Deletes stored log records older than this threshold.

External collectors such as SIEM platforms, syslog receivers, HTTP collectors, and data lakes have independent retention settings.

Stored log cleanup is off by default. The 180 day value applies only after both the cleanup scheduler and stored log cleanup are enabled.

Log retention cleanup uses the same cleanup cron and timezone settings as hard delete cleanup. Review Hard delete cron configuration before changing the shared cleanup schedule.

Syslog delivery

Syslog delivery sends formatted log records to a socket collector over TCP, TLS, or UDP. Use rfc5424_json for structured syslog pipelines. Use cef when the collector has a native CEF input.

Format	Description	Use when
`rfc5424_json`	RFC 5424 syslog with a JSON envelope in the message body. The schema value is `zylon.rfc5424_json.v1`.	You want structured syslog events for SIEM/search pipelines.
`cef`	Raw CEF event format. The CEF extension includes a raw structured record under `cs1`.	Your collector has a native CEF input.

Transport	Behavior
`tls`	Persistent TLS socket. Supports certificate verification and custom CA file.
`tcp`	Persistent TCP socket without TLS.
`udp`	Datagram delivery. One event is sent per datagram; stream framing is ignored.

UDP is best-effort. It does not use stream framing and does not retry the current batch on write failure.

Supported syslog matrix

Backend tests confirm both syslog formats with tcp, tls, and udp.

Format	Transport	Framing	Notes
`rfc5424_json`	`tls`	`newline` or `octet_counting`	Recommended secure path for structured syslog collectors.
`rfc5424_json`	`tcp`	`newline` or `octet_counting`	Persistent socket path without TLS.
`rfc5424_json`	`udp`	Datagram	One datagram per event.
`cef`	`tcp`	`newline` or `octet_counting`	Raw CEF over TCP.
`cef`	`tls`	`newline` or `octet_counting`	Raw CEF over TLS.
`cef`	`udp`	Datagram	One datagram per event.

Syslog keys

Under logging.delivery.syslog:

Key	Default	Values	Description
`enabled`	`false`	`true`, `false`	Enables syslog delivery.
`format`	`rfc5424_json`	`rfc5424_json`, `cef`	Syslog payload format.
`host`	empty	hostname or IP	Required collector host.
`port`	`6514`	valid TCP/UDP port	Collector port.
`transport`	`tls`	`tls`, `tcp`, `udp`	Socket transport.
`framing`	`newline`	`newline`, `octet_counting`	Stream framing for TCP/TLS. Ignored for UDP.

Optional syslog keys

Optional filter, delivery, queue, batch, and TLS settings are configured under their matching nested blocks in logging.delivery.syslog.Under filters:

Key	Default	Values	Description
`includeActionPatterns`	`[]`	wildcard list	If non-empty, only matching action names are delivered.
`excludeActionPatterns`	`[]`	wildcard list	Matching action names are excluded before include filters are evaluated.
`includeOrgIds`	`[]`	organization UUID list	If non-empty, only logs for matching organization IDs are delivered.

Under delivery:

Key	Default	Values	Description
`timeoutSeconds`	`10`	positive integer	Socket connect/write timeout.
`failurePolicy`	`drop`	`drop`, `block`	Queue behavior when the async queue is full.

Under delivery.reconnect:

Key	Default	Values	Description
`enabled`	`true`	`true`, `false`	Enables reconnect after connect or write failure.
`backoffSeconds`	`5`	non-negative integer	Backoff before reconnect after failure.

Under delivery.queue:

Key	Default	Values	Description
`maxEvents`	`10000`	positive integer	Async delivery queue capacity.

Under delivery.batch:

Key	Default	Values	Description
`maxEvents`	`25`	positive integer	Maximum events per batch. Forced to `1` for UDP.
`maxDelayMillis`	`250`	positive integer	Maximum time to wait before flushing a partial batch.
`drainTimeoutMillis`	`1000`	positive integer	Shutdown drain timeout for queued batch entries.

Under tls:

Key	Default	Values	Description
`verifyCertificate`	`false`	`true`, `false`	Enables TLS certificate and hostname verification.
`caCrt`	empty	PEM certificate data	Inline CA certificate data. If set and `caFile` is empty, the chart mounts it for the backend.
`caFile`	empty	readable file path	Custom CA bundle used when certificate verification is enabled.

Syslog full example

logging:
  storage:
    enabled: true
  delivery:
    syslog:
      enabled: true
      format: "rfc5424_json"
      endpoint:
        host: "syslog.example.com"
        port: 6514
        transport: "tls"
        framing: "octet_counting"
      filters:
        includeActionPatterns:
          - "ws.*"
          - "backoffice.*"
        excludeActionPatterns:
          - "ws.user.enable"
        includeOrgIds:
          - "01985a4d-3c21-7000-8003-0e10dd81d901"
      delivery:
        timeoutSeconds: 10
        failurePolicy: "drop"
        reconnect:
          enabled: true
          backoffSeconds: 5
        queue:
          maxEvents: 10000
        batch:
          maxEvents: 25
          maxDelayMillis: 250
          drainTimeoutMillis: 1000
      tls:
        verifyCertificate: true
        caFile: "/etc/zylon/certs/syslog-ca.pem"

Syslog simplified example

logging:
  delivery:
    syslog:
      enabled: true
      format: "rfc5424_json"
      endpoint:
        host: "syslog.example.com"
        port: 6514
        transport: "tls"

HTTP delivery

HTTP delivery sends structured JSON records to an HTTP collector. Use it for ingestion gateways, data lakes, or pipelines that transform Zylon’s canonical payload downstream. The supported HTTP format is canonical_json. The schema value is zylon.canonical_json.v1. HTTP supports POST, optional bearer authentication, static headers using Header=Value or Header: Value, retries for network errors and HTTP 5xx responses, and no retry for HTTP 4xx client errors. Vendor-specific transformations should happen at the collector or ingestion layer.

HTTP keys

Under logging.delivery.http:

Key	Default	Values	Description
`enabled`	`false`	`true`, `false`	Enables HTTP delivery.
`format`	`canonical_json`	`canonical_json`	HTTP payload format.
`url`	empty	URL	Required when HTTP delivery is enabled.
`method`	`POST`	`POST`	HTTP method.
`headers`	`[]`	YAML list	Static outbound headers. Supports `Header=Value` and `Header: Value`.

Optional HTTP keys

Optional authentication, filter, delivery, queue, and batch settings are configured under their matching nested blocks in logging.delivery.http.Under auth:

Key	Default	Values	Description
`type`	`none`	`none`, `bearer`	Authentication mode.
`token`	empty	string	Required when `type` is `bearer`. Sent as `Authorization: Bearer <token>`.

Under filters:

Key	Default	Values	Description
`includeActionPatterns`	`[]`	wildcard list	If non-empty, only matching action names are delivered.
`excludeActionPatterns`	`[]`	wildcard list	Matching action names are excluded before include filters are evaluated.
`includeOrgIds`	`[]`	organization UUID list	If non-empty, only logs for matching organization IDs are delivered.

Under delivery:

Key	Default	Values	Description
`timeoutSeconds`	`10`	positive integer	HTTP connect, read, and write timeout.
`failurePolicy`	`drop`	`drop`, `block`	Queue behavior when the async queue is full.

Under delivery.retry:

Key	Default	Values	Description
`enabled`	`true`	`true`, `false`	Enables HTTP retry.
`maxAttempts`	`3`	positive integer	Maximum request attempts when retry is enabled.
`backoffSeconds`	`3`	non-negative integer	Sleep between retry attempts.

Under delivery.queue:

Key	Default	Values	Description
`maxEvents`	`10000`	positive integer	Async delivery queue capacity.

Under delivery.batch:

Key	Default	Values	Description
`maxEvents`	`25`	positive integer	Maximum events per HTTP request. Multi-event batches are sent as a JSON array.
`maxDelayMillis`	`250`	positive integer	Maximum time to wait before flushing a partial batch.
`drainTimeoutMillis`	`1000`	positive integer	Shutdown drain timeout for queued batch entries.

HTTP full example

logging:
  delivery:
    http:
      enabled: true
      format: "canonical_json"
      endpoint:
        url: "https://collector.example.com/logs"
        method: "POST"
      auth:
        type: "bearer"
        token: "replace-me"
      headers:
        - "X-Collector-Token=replace-me"
        - "X-Customer: default"
      filters:
        includeActionPatterns:
          - "zylon.gateway.*"
          - "ws.*"
        excludeActionPatterns: []
        includeOrgIds: []
      delivery:
        timeoutSeconds: 10
        failurePolicy: "drop"
        retry:
          enabled: true
          maxAttempts: 3
          backoffSeconds: 3
        queue:
          maxEvents: 10000
        batch:
          maxEvents: 25
          maxDelayMillis: 250
          drainTimeoutMillis: 1000

HTTP simplified example

logging:
  delivery:
    http:
      enabled: true
      format: "canonical_json"
      endpoint:
        url: "https://collector.example.com/logs"
        method: "POST"

Filtering delivered events

Syslog and HTTP delivery use the same filter behavior under filters.

Key	Default	Values	Description
`includeActionPatterns`	`[]`	wildcard list	If non-empty, only matching action names are delivered. `*` matches any number of characters. Matching is case-sensitive and covers the full action string.
`excludeActionPatterns`	`[]`	wildcard list	Matching action names are excluded before include filters are evaluated.
`includeOrgIds`	`[]`	organization UUID list	If non-empty, only logs for matching organization IDs are delivered. Logs without `organization_id` are not delivered.

Evaluation order:

Exclude action patterns.
Include action patterns.
Included organization IDs.

Delivery behavior

The delivery pipeline is:

Zylon emits a structured log event.
Delivery filters decide whether to deliver it externally.
The event is normalized with severity, outcome, and actor metadata.
The event is formatted as syslog, CEF, or canonical JSON.
The formatted payload is placed on the async delivery queue.
The delivery worker sends the event to the configured destination.
Reconnect or retry behavior applies when delivery fails.

Operational controls

Keep all Zylon backend nodes synchronized to a trusted time source such as NTP. The Helm chart sets the cleanup scheduler timezone to UTC by default; if the runtime setting is omitted outside Helm, the backend uses the server JVM time zone. Log records should be reviewed in UTC for cross-system forensic correlation. Treat changes to logging.storage, logging.delivery, filters, retention, and collector destinations as auditable deployment changes. Capture approvals and diffs in GitOps, change management, Kubernetes audit logs, or the infrastructure audit system used by the customer. Delivered payload schemas use markers such as zylon.rfc5424_json.v1 and zylon.canonical_json.v1. SIEM parsers should route by schema marker and validate parser compatibility during upgrades.

Failure policies

Policy	Behavior	Use when
`drop`	Drops events when the async queue is full and avoids blocking request threads.	Recommended default for most production deployments.
`block`	Applies backpressure when the queue is full.	Use only when delivery completeness is more important than request latency.

failurePolicy: "block" can slow requests when the delivery queue is full. Use it only after sizing queues and confirming collector availability.

Disk-backed external delivery spooling is not implemented.

Queue and batching

Setting	Description
`maxEvents`	Maximum events that can wait in the async delivery queue, or maximum events per delivery batch when set under `batch`. Larger queues absorb bursts but use more memory; larger batches can improve throughput but can increase delay.
`maxDelayMillis`	Maximum time before a partial batch is flushed. Smaller values reduce latency.
`drainTimeoutMillis`	Time allowed to drain queued batch entries during shutdown.

For syslog over UDP, the maxEvents value under delivery.batch is forced to 1, because one datagram carries one event. For HTTP, multi-event batches are delivered as a JSON array.

Queue and batching settings already have production defaults. If your HTTP destination, SIEM, or log collector does not require specific values, leave the defaults unchanged. Customize them only when your environment needs different throughput, latency, or burst handling.

Reconnect and retry

Syslog reconnect

TCP and TLS syslog delivery lazily opens a socket, reuses it, and retries the current batch once on a fresh connection after write failure. If retry fails, the reconnect backoff window applies.

Setting	Description
`enabled`	Enables reconnect after socket failure. If disabled, future reconnect attempts stop after the first failure until process restart or appender reset.
`backoffSeconds`	Time before another connect attempt after failure.

HTTP retry

HTTP delivery reuses one HTTP client and retries failed requests according to retry settings.

Setting	Description
`enabled`	Enables retry after retryable failures.
`maxAttempts`	Maximum attempts for one delivery.
`backoffSeconds`	Sleep between attempts.

HTTP retries network exceptions and HTTP 5xx responses. HTTP 4xx responses are treated as client errors and are not retried.

TLS configuration

Setting	Description
`verifyCertificate`	Enables TLS certificate verification and hostname verification for syslog TLS.
`caFile`	Path to a readable CA file used to trust a private collector CA when verification is enabled.

Local or lab deployments may disable certificate verification. Production deployments should enable certificate verification when the collector has a trusted certificate path. Use caFile under tls when the collector uses a private CA.

Local validation

Enable one delivery channel.
Apply configuration.
Trigger an API action that emits a log, such as project creation or API token creation.
Query the collector.
Confirm the expected event, action, log.id, and actor fields appear.
Check backend logs for delivery warnings.

Validation queries depend on your collector. Use whichever search or inspection tool your destination provides, then confirm one of these payload markers appears:

zylon.rfc5424_json.v1 for RFC 5424 JSON syslog payloads.
CEF:0|Zylon for CEF payloads.
zylon.canonical_json.v1 for HTTP payloads.
A known action such as ws.project.create.

For reconnect testing, stop the collector, trigger events, restart the collector, then confirm new events arrive after the reconnect backoff.

Troubleshooting

Symptom	What to check
No events arrive	`enabled`, endpoint host or URL, port, collector reachability, and backend logs.
Syslog startup fails	Missing host, invalid port, unsupported format/transport/framing, or unreadable CA file.
TLS connection fails	Certificate verification, CA file, collector certificate, and hostname.
CEF events are not parsed	`format: "cef"`, collector input type, and port.
Requests slow down	`failurePolicy: "block"`, queue size, collector throughput, and backend logs.
Missing events during bursts	Queue size, batch settings, collector throughput, and whether `failurePolicy: "drop"` is acceptable.
Events do not reconnect after outage	Reconnect enabled setting, backoff setting, and backend logs.
HTTP delivery fails	Endpoint URL, method, bearer token, static headers, retry settings, and backend logs.
HTTP keeps failing without retry	Collector HTTP `4xx` responses. Fix auth, URL, headers, or collector route.
Retention is shorter than expected	Log cleanup settings, DB retention/backup policy, and downstream collector retention.

Recommended production defaults

For syslog and SIEM:

Prefer TLS where supported.
Prefer rfc5424_json for structured syslog ingestion unless the collector requires CEF.
Use octet_counting or newline according to the collector input.
Enable certificate verification in production where possible.
Keep failurePolicy: "drop" unless strict delivery completeness is more important than request latency.
Keep queue and batch defaults unless your destination requires different throughput, latency, or burst handling.
Keep stored log retention at or above 180 days unless legal, security, and operations teams approve a shorter period.

For HTTP:

Use canonical_json.
Keep vendor-specific transformations at the ingestion layer.
Use bearer authentication and static headers when required by the collector.
Tune retry settings for the collector’s availability profile.
Confirm downstream retention in the customer-controlled collector.

​Overview

​At a glance

​Before you configure delivery

​Choose a delivery channel

​Storage and retention

​Storage keys

​Retention keys

​Syslog delivery

​Supported syslog matrix

​Syslog keys

​Syslog full example

​Syslog simplified example

​HTTP delivery

​HTTP keys

​HTTP full example

​HTTP simplified example

​Filtering delivered events

​Delivery behavior

​Operational controls

​Failure policies

​Queue and batching

​Reconnect and retry

​Syslog reconnect

​HTTP retry

​TLS configuration

​Local validation

​Troubleshooting

​Recommended production defaults

Overview

At a glance

Before you configure delivery

Choose a delivery channel

Storage and retention

Storage keys

Retention keys

Syslog delivery

Supported syslog matrix

Syslog keys

Syslog full example

Syslog simplified example

HTTP delivery

HTTP keys

HTTP full example

HTTP simplified example

Filtering delivered events

Delivery behavior

Operational controls

Failure policies

Queue and batching

Reconnect and retry

Syslog reconnect

HTTP retry

TLS configuration

Local validation

Troubleshooting

Recommended production defaults