Secure Syslog and CEF Logging

Overview

Syslog and CEF are two widely used standards for logging and event management in the field of IT and network security.

Syslog is a standard protocol used for sending and receiving log and event messages between network devices, servers, and applications. It provides a structured and standardized way of capturing and forwarding system messages, allowing IT administrators to monitor and troubleshoot network and system issues more effectively. Syslog messages can contain various types of information, including system events, errors, and warnings, as well as user activity and other operational data.

CEF (Common Event Format) is a standardized log format for capturing and transmitting security-related events across various systems and devices. CEF provides a common language and format for security information and event management (SIEM) systems, allowing for more efficient and effective analysis of security events and threats.

See the Syslog Event List for a complete list of events.

Secret Server and Syslog or CEF Logging

Secret Server can send a copy of important log messages to an external syslog server for added security using the following protocols:

Common Event Format (CEF) is an industry-standard format on top of syslog messages that ensures event interoperability between different platforms.

Table: Syslog Transportation Protocols

Protocol	Encrypted	Notes
UDP	No	Least reliable. User Datagram Protocol (UDP) traffic is fire-and-forget with no assurance messages are delivered and no error checking.
TCP	No	More reliable. Transmission Control Protocol (TCP) ensures messages arrive in order, missing messages are resent, and has built in error checking.
Secure TCP	Yes	Establishes a secure connection — Transport Layer Security (TLS) 1.1 or 1.2 only. Syslog Server's certificate is validated by Windows to ensure it is trusted and not revoked. Can be used with or without client certificates (configured in Configuration > Security tab > TLS Auditing > Advanced).

Due to the sensitive nature of Secret Server logs, we strongly recommend using Secure TCP.

Configuring a Secure TCP Syslog or CEF External Audit Server in Secret Server

Compatible Audit Servers

syslog-ng
Any Audit server that accepts TLS encrypted messages using the BSD syslog protocol

Special Characters in Syslogs

Secret Server uses UTF-8 encoding for syslog messages, which supports non-ASCII characters. Some syslog servers may only support ASCII. If you cannot change your server to UTF-8, your syslog messages may include error characters (often question marks) where non-ASCII characters appear.

Configuring an External Audit Server

Navigate to Admin > Application. The Application page appears.
Click the Edit button at the top right of the page.
Click to select the Enable Syslog/CEF Log Output check box. A syslog/CEF section expands below.

Note: syslog/CEF may require an additional license key. To install licenses, navigate to Admin > Licenses > Install New License. Once installed, the license requires activation. Contact your Delinea Sales Representative with any questions.
Type IP address or name for the IIS server hosting the syslog/CEF server in the Syslog/CEF Server text box.

You can add multiple entries, separating each with a semicolon.
Type the port number where the logging information will be passed (6514 is the default port for secure TCP syslog) in the Syslog/CEF Port text box.

Note: Secret Server requires outbound access to this server and port so communication can pass freely.
Click the Syslog/CEF Protocol dropdown list and select Secure TCP. Secure TCP means either TLS v1.2 or v1.1 because other versions of SSL, such as SSL v3 and TLS v1.0, have known weaknesses.
Click to select Syslog/CEF Time Zone list box to UTC Time or Server Time, depending on your preference.
Click the Syslog/CEF DateTime Format dropdown to format Syslog timestamps. The standard for Syslog is ISO timestamps; however, some still use the legacy format. Syslog is the default for upgrades to allow current configurations to retain their behavior, and ISO format is the default in new instances. Syslog format: Jun 23 2022 11:22:33. ISO 8601 format: 2022-06-23T11:22:33.000. You must enable the configuration preview to modify this setting.
Click the Syslog/CEF Site dropdown to select the related Site that Syslog/CEF will run on.
Click to select the Write Syslogs As Windows Events check box to write audits and event subscriptions to the Windows Event Log of the server.
Click the Save button.

Caching Syslog Audits and the Syslog Circuit Breaker

Enabling Secure Syslog Logging

Once secure syslog logging is enabled in Secret Server, ensure that the system is set to cache syslog failure notification messages in the Secret Server database if the connection to the external syslog server breaks.

Configuring Syslog Server for Non-Local Sites

If the syslog server uses a non-local site, implement a circuit breaker system to avoid overwhelming the distributed engine and message queues during likely failure scenarios.

Understanding Circuit Breaker States

The circuit breaker enters the "open" state after ten failures within five minutes.
In the "open" state, the system will not attempt to send syslog messages.
Every five minutes, the circuit breaker shifts to the "half-open" state to attempt recovery by sending a limited number of syslog messages.
Successful delivery of three syslog messages transitions the circuit breaker to the "closed" state, resuming normal syslog message processing.

Monitoring System Alerts

When the circuit breaker is in "open" or "half-open" mode, administrators will receive a notification banner. This banner provides a link to the system log for accessing diagnostic messages, including details on which engines had issues sending messages to the syslog server and other diagnostic information.

Configure Auditing for TLS Connections

To track problems with TLS connections (including whenever the connection fails), enable the TLS certificate chain policy and error auditing in Secret Server:

Navigate to Admin > Configuration.
Click the Security tab.
Click the Edit button at the bottom of the page.
Scroll to the TLS Auditing section.
Ensure the Apply TLS Certificate Chain Policy and Error Auditing check box is enabled. If not, you cannot use client certificates.

If secure TCP is used for the syslog/CEF protocol and there are one or more client certificate thumbprints entered, Secret Server checks the local computer's Web hosting and personal certificate store and uses the first one it finds.

Adding Client Certificate Thumbprints

Navigate to Admin > Configuration.
Click the Security tab.
Click the Edit button at the bottom of the page.
Scroll to the TLS Auditing section.
Click the Advances (not required) link. A client certificate thumbprint section appears.
Copy and paste a list of SHA1 SSL certificate thumbprints into the Client Certificate Thumbprints(s) text box. Separate each thumbprint (40 characters each) with a semicolon. Up to ten are allowed.

Secret Server's IIS application pool must be granted permission to use the client certificates, using the Windows HTTP Services Certificate Configuration Tool (WinHttpCertCfg.exe). See Compatibility Notes for Client Certificates.

Determining the Status of a Remote Audit Server

To view the logs for any TLS-Connection related errors, perform the following:

Open the Microsoft SQL Server Management Studio.
Navigate to your SecretServer database at <DB Machine Name> > Databases > SecretServer).
Set up a new query.
Type and enter select from tbSecurityAuditLog to view the events from the TLS audit.

For more detailed troubleshooting reporting, reference the logs on the Secret Server Web server at C:\inetpub\wwwroot\SecretServer\log). View the SS.log, SS-BSSR.log (background scheduler), and SS-BSWR.log (background worker) for any errors.

Compatibility Notes for Client Certificates

IIS Application Pool Certificate Permissions

Secret Server's IIS application pool must be granted permission to use the client certificates, using the Windows HTTP Services Certificate Configuration Tool (WinHttpCertCfg.exe).

For example: winhttpcertcfg.exe -g -c LOCAL_MACHINE\MY -s "Certificate Subject" -a "HOSTNAME\IIS APPPOOL\SecretServer"

You can download the tool at:

Windows HTTP Services Certificate Configuration Tool (WinHttpCertCfg.exe)

You can view the documentation at:

WinHttpCertCfg.exe, a Certificate Configuration Tool

Otherwise, if Secret Server is configured to use a client certificate, and IIS does not have permission, errors like this may appear in the logs:

TLS Error Detected (Authentication Error connecting to IP:PORT) - The credentials supplied to the package were not recognized.

If you are using a client certificate, and a syslog-ng logging server, the following message may occasionally appear in the main syslog-NG log file:

SSL error while reading stream; tls_error='SSL routines:ssl_get_prev_session:session id context uninitialized'

On the Secret Server side, this appears:

TLS Error Detected (Authentication Error connecting to IP:PORT) - Authentication failed because the remote party has closed the transport stream.

This is caused by Windows trying to cache secure connections when client certificates are used, but because syslog-ng has not configured the OpenSSL "session id context", OpenSSL displays this error when it tries to resume a previous session.

Secret Server automatically reconnects and resends any missed messages, so the errors should not have an impact. However, you can disable Window's secure connection caching by adding the ClientCacheTime setting set to 0 in the Registry and then rebooting. This did not cause any significant performance impact in internal testing.

If changing back to a previous syslog IP address and port, you will receive a closed connection TLS error on the first attempted syslog connection after making the change. A subsequent call will succeed as the first failure will clear the cached connection on Windows. This is due to the issue with syslog-ng.

If syslog-ng configures their OpenSSL session id context, this error message correction is no longer needed.

AlienVault

It is common for people to incorrectly use the client certificate thumbprints feature when setting up secure AlienVault for syslog. This can cause Secret Server to try to connect to LDAPS with the AlienVault certificate, which can break LDAPS. Users should not use the Secret Server client certificates thumbprint for specifying one certificate for syslog and another for LDAP. The certificate list is intended for each Secret Server or DE to have its own, unique certificate.