SPLK-1003 Splunk Enterprise Certified Admin Questions and Answers

The following items in the phases below are listed in the order Splunk applies them (ie LINE_BREAKER occurs before TRUNCATE).

Input phase

inputs.conf

props.conf

CHARSET

NO_BINARY_CHECK

CHECK_METHOD

CHECK_FOR_HEADER (deprecated)

PREFIX_SOURCETYPE

sourcetype

wmi.conf

regmon-filters.conf

Structured parsing phase

props.conf

INDEXED_EXTRACTIONS, and all other structured data header extractions

Parsing phase

props.conf

LINE_BREAKER, TRUNCATE, SHOULD_LINEMERGE, BREAK_ONLY_BEFORE_DATE, and all other line merging settings

TIME_PREFIX, TIME_FORMAT, DATETIME_CONFIG (datetime.xml), TZ, and all other time extraction settings and rules

TRANSFORMS which includes per-event queue filtering, per-event index assignment, per-event routing

SEDCMD

MORE_THAN, LESS_THAN

transforms.conf

stanzas referenced by a TRANSFORMS clause in props.conf

LOOKAHEAD, DEST_KEY, WRITE_META, DEFAULT_VALUE, REPEAT_MATCH

https://docs.splunk.com/Documentation/Splunk/7.3.1/DistSearch/Howuserscancontroldistributedsearches

"From the user standpoint, specifying and running a distributed search is essentially the same as running any other search. Behind the scenes, the search head distributes the query to its search peers, and consolidates the results when presenting them to the user."

The Universal Forwarder (UF) handles data forwarding to indexers. When monitoring a continuous and high-volume syslog stream over TCP, the UF may not detect an end-of-file (EOF) condition, which is typicallyrequired to trigger load balancing between multiple indexers. This can result in the UF continuously sending data to a single indexer, potentially leading to uneven load distribution.

To address this, Splunk provides the forceTimebasedAutoLB setting in the outputs.conf configuration file. Enabling this setting allows the UF to switch between indexers at regular time intervals, regardless of EOF detection. This ensures a more balanced distribution of data across multiple indexers, even in scenarios with continuous data streams like syslog.

https://docs.splunk.com/Documentation/Splunk/7.3.0/Admin/Wheretofindtheconfigurationfiles

"To determine the order of directories for evaluating configuration file precendence, Splunk software considers each file's context. Configuration files operate in either a global context or in the context of the current app and user"

https://docs.splunk.com/Splexicon:Serverclass

You can have a role inherit certain properties from one or more existing rolehttps://docs.splunk.com/Documentation/Splunk/8.0.5/Security/Aboutusersandroles

When there are conflicting settings within two or more configuration files, the setting with the highest precedence is used. The precedence of configuration files is determined by a combination of the file type, the directory location, and the alphabetical order of the file names.

This option corresponds to the file path “$SPLUNK_HOME/etc/apps/splunk_TA_nginx/local/inputs.conf”. This is the configuration file that the user needs to edit to ingest the NGINX access logs to ensure it remains unaffected after upgrade. This is explained in the Splunk documentation, which states:

The local directory is where you place your customized configuration files. The local directory is empty when you install Splunk Enterprise. You create it when you need to override or add to the default settings in a configuration file. The local directory is never overwritten during an upgrade.

Setting: ignoreOlderThan = Description: "Causes the input to stop checking files for updates if the file modification time has passed the threshold." Default: 0 (disabled)

To verify the integrity of a local bucket. The command ./splunk check-integrity -bucketPath [bucket path] [-verbose] is used to verify the integrity of a local bucket by comparing the hashes stored in the l1Hashes and l2Hash files with the actual data in the bucket1. This command can help detect any tampering or corruption of the data.

According to the Splunk documentation1, a heavy forwarder is a Splunk Enterprise instance that can parse and filter data before forwarding it to an indexer. A heavy forwarder can perform line breaking,which is the process of splitting incoming data into individual events based on a set of rules2. A heavy forwarder can also apply other transformations to the data, such as field extractions, event type matching, or masking sensitive data3.

Referencehttps://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/Configurationparametersandthedatapipeline

 The parsing phase is the process of extracting fields and values from raw data. The parsing phase respects LINE_BREAKER, SHOULD_LINEMERGE, BREAK_ONLY_BEFORE_DATE, and all other line merging settings in props.conf. These settings determine how Splunk breaks the data into events based on certain criteria, such as timestamps or regular expressions. The event boundaries are defined by the props.conf file, which can be modified by the administrator. Therefore, the parsing phase is the last opportunity for defining event boundaries.

https://docs.splunk.com/Documentation/Splunk/8.1.1/DistSearch/Forwardsearchheaddata

Per the provided Splunk reference URL by @hwangho, scroll to section Forward search head data, subsection titled, 2. Configure the search head as a forwarder. "Create an outputs.conf file on the search head that configures the search head for load-balanced forwarding across the set of search peers (indexers)."

The correct answer is C. MonitorNoHandle.

MonitorNoHandle is a type of input stanza that allows a Splunk forwarder to read files on Windows systems as Windows writes to them. It does this by using a kernel-mode filter driver to capture raw data as it gets written to the file1. This input stanza is useful for files that get locked open for writing, such as the Windows DNS server log file2.

The other options are incorrect because:

A. Tail Reader is not a valid input stanza in Splunk. It is a component of the Tailing Processor, which is responsible for monitoring files and directories for new data3.

B. Upload is a type of input stanza that allows Splunk to index a single file from a local or network file system. It is not suitable for files that are constantly being updated, as it only indexes the file once and does not monitor it for changes4.

D. Monitor is a type of input stanza that allows Splunk to monitor files and directories for new data. However, it may not work for files that Windows prevents Splunk from reading while they are open. In such cases, MonitorNoHandle is a better option2.

A Splunk forwarder is a lightweight agent that can forward data to a Splunk deployment. There are two types of forwarders: universal and heavy. A universal forwarder can only forward data, while a heavy forwarder can also perform parsing, filtering, routing, and aggregation on the data before forwarding it5.

An input stanza is a section in the inputs.conf configuration file that defines the settings for a specific type of input, such as files, directories, network ports, scripts, or Windows event logs. An input stanza starts with a square bracket, followed by the input type and the input path or name. For example, [monitor:///var/log] is an input stanza for monitoring the /var/log directory.

In Splunk Enterprise, when you configure the server.conf file with strict pool quota=false, it means that license pools are allowed to share the total available license quota rather than being restricted to their individually allocated quotas. However, this does not prevent pools from issuing warnings if they exceed their allocated limits.

Given the environment with a 1000 GB/day license split into three pools:

Pool X: 500 GB/day license, 100 GB used

Pool Y: 350 GB/day license, 400 GB used

Pool Z: 150 GB/day license, 300 GB used

Let's analyze the usage:

Pool X is allocated 500 GB/day but has only used 100 GB, well within its limit.

Pool Y is allocated 350 GB/day but has used 400 GB, which exceeds its limit by 50 GB.

Pool Z is allocated 150 GB/day but has used 300 GB, which exceeds its limit by 150 GB.

Even with strict pool quota=false, pools Y and Z have exceeded their individual allocated quotas and will issue warnings. Pool X has not exceeded its quota and thus will not issue any warnings. Therefore, the pools that are issued warnings are Y and Z.

Quoting the following Splunk URL reference https://docs.splunk.com/Documentation/Splunk/8.2.2/DMC/DMCprerequisites "Monitoring Console setup prerequisites. Forward internal logs (both $SPLUNK_HOME/car/log/splunk and $SPLUNK_HOME/var/log/introspection) to indexers from all other components. Without this step, many dashboards will lack data."

https://docs.splunk.com/Documentation/Splunk/latest/Data/Configureeventlinebreaking

Attribute : SHOULD_LINEMERGE = [true|false]

Description : When set to true, the Splunk platform combines several input lines into a single event, with configuration based on the settings described in the next section.

According to the Splunk system admin course PDF, When adding native users, Username and Password ARE REQUIRED

https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/Configureyourinputs

Add your data to Splunk Enterprise. With Splunk Enterprise, you can add data using Splunk Web or Splunk Apps. In addition to these methods, you also can use the following methods. -The Splunk Command Line Interface (CLI) -The inputs.conf configuration file. When you specify your inputs with Splunk Web or the CLI, the details are saved in a configuartion file on Splunk Enterprise indexer and heavy forwarder instances.

In Splunk Enterprise and Splunk Universal Forwarder, data inputs are configured using stanzas in inputs.conf. Each stanza defines a listener for a particular input type (for example, TCP or UDP) and a specific port.

Splunk documentation states that a single TCP input stanza can receive data from multiple remote hosts sending to the same TCP port, and similarly, a single UDP input stanza can receive data from multiple devices sending to the same UDP port. Therefore, the number of sending devices does not determine the number of stanzas required; rather, the input protocol and port type do.

In this case:

All three TCP devices can send data to one TCP port (one stanza).

All four UDP devices can send data to one UDP port (one stanza).

Thus, the minimum number of input stanzas required is two — one for TCP and one for UDP.

Example configuration (inputs.conf):

# TCP input for three switches sending via TCP

[tcp://9997]

sourcetype = switch_logs

# UDP input for four switches sending via UDP

[udp://514]

sourcetype = switch_logs

This configuration ensures all seven devices’ logs are collected without creating individual stanzas for each device.

Reference (Splunk Documentation):

Splunk® Enterprise Admin Manual → Configure Data Inputs → “Listen for network data”

inputs.conf.spec and example → “You can configure a single TCP or UDP input to receive data from multiple remote hosts.”

Splunk Universal Forwarder Manual → Configure Forwarding Inputs → “Universal Forwarders can listen on a single TCP or UDP port for multiple remote data sources.”

"Where to place the scripts for scripted inputs. The script that you refer to in $SCRIPT can reside in only one of the following places on the host file system:

$SPLUNK_HOME/etc/system/bin

$SPLUNK_HOME/etc/apps//bin

$SPLUNK_HOME/bin/scripts

As a best practice, put your script in the bin/ directory that is nearest to the inputs.conf file that calls your script on the host file system."

https://docs.splunk.com/Documentation/Splunk/8.1.1/Data/Monitornetworkports

https://docs.splunk.com/Documentation/Splunk/latest/Data/Whitelistorblacklistspecificincomingdata#Include_or_exclude_specific_incoming_data

https://docs.splunk.com/Documentation/Splunk/8.0.1/Troubleshooting/Usebtooltotroubleshootconfigurations

"The btool command simulates the merging process using the on-disk conf files and creates a report showing the merged settings."

"The report does not necessarily represent what's loaded in memory. If a conf file change is made that requires a service restart, the btool report shows the change even though that change isn't active."

The Data Preview feature should be used when validating the parsing of data. The Data Preview feature allows you to preview how Splunk software will index your data before you commit the data to an index. You can use the Data Preview feature to check the following aspects of data parsing1:

Timestamp recognition: You can verify that Splunk software correctly identifies the timestamps of your events and assigns them to the _time field.

Event breaking: You can verify that Splunk software correctly breaks your data stream into individual events based on the line breaker and should linemerge settings.

Source type assignment: You can verify that Splunk software correctly assigns a source type to your data based on the props.conf file settings. You can also manually override the source type if needed.

Field extraction: You can verify that Splunk software correctly extracts fields from your events based on the transforms.conf file settings. You can also use the Interactive Field Extractor (IFX) to create custom field extractions.

The Data Preview feature is available in Splunk Web under Settings > Data inputs > Data preview. You can access the Data Preview feature when you add a new input or edit an existing input1.

The other options are incorrect because:

A. When extracting fields for ingested data. The Data Preview feature can be used to verify the field extraction for data that has not been ingested yet, but not for data that has already been indexed. To extract fields from ingested data, you can use the IFX or the rex command in the Search app2.

B. When previewing the data before searching. The Data Preview feature does not allow you to search the data, but only to view how it will be indexed. To preview the data before searching, you can use the Search app and specify a time range or a sample ratio.

C. When reviewing data on the source host. The Data Preview feature does not access the data on the source host, but only the data that has been uploaded or monitored by Splunk software. To review data on the source host, you can use the Splunk Universal Forwarder or the Splunk Add-on for Unix and Linux.

In Splunk Enterprise, license warnings occur when the daily indexing volume exceeds the licensed quota. These warnings are tracked from midnight to midnight based on the system clock of the license manager. If the number of warnings surpasses the allowed threshold within a specified period, a license violation ensues, potentially restricting search capabilities.

To prevent a license warning from escalating to a violation, administrators have until midnight to address the issue. The recommended action is toadd a new licenseto the license manager before midnight. This increases the daily indexing volume quota, ensuring that the current day's data ingestion falls within the permissible limits.

It's important to note that deleting data from indexers or the license manager does not retroactively reduce the recorded license usage for the day. Once data is indexed, it contributes to the day's license volume, and its removal does not negate that contribution.

Adding further clarity and quoting same Splunk reference URL from @giubal"

"To keep configuration settings consistent across peer nodes, configuration files are managed from the cluster master, which pushes the files to the slave-app directories on the peer nodes. Files in the slave-app directories have the highest precedence in a cluster peer's configuration. Here is the expanded precedence order for cluster peers:

1.Slave-app local directories -- highest priority

2. System local directory

3. App local directories

4. Slave-app default directories

5. App default directories

6. System default directory --lowest priority

https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Protectagainstlossofin-flightdata#:~:text=The%20default%20for%20the%20maxQueueSize,wait%20queue%20size%20is%2021MB.

https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Protectagainstlossofin-flightdata

A role in Splunk is a classification that determines what capabilities and indexes a user has.A capability is a permission to perform a specific action or access a specific feature on the Splunk platform1.An index is a collection of data that Splunk software processes andstores2. By assigning roles to users, you can control what they can do and what data they can access on the Splunk platform.

Therefore, the correct answers are A and D. A role in Splunk determines what capabilities and indexes a user has. Option B is incorrect because Splunk servers do not use roles to remotely control each other.Option C is incorrect because Splunk servers use instances and components to determine what functions they control3.

SAML is an open standard for exchanging authentication and authorization data between parties, especially between an identity provider and a service provider1. SAML supports multi-factor authentication by allowing the identity provider to require the user to present two or more factors of evidence to prove their identity2. For example, the user may need to enter a password and a one-time code sent to their phone, or scan their fingerprint and face.

"The search head replicates the knowledge bundle periodically in the background or when initiating a search. " "As part of the distributed search process, the search head replicates and distributes its knowledge objects to its search peers, or indexers. Knowledge objects include saved searches, event types, and other entities used in searching accorss indexes. The search head needs to distribute this material to its search peers so that they can properly execute queries on its behalf."

The correct answer is C and D. A heavy forwarder and an indexer are the Splunk components that can break a stream of syslog inputs into individual events.

A universal forwarder is a lightweight agent that can forward data to a Splunk deployment, but it does not perform any parsing or indexing on the data. A search head is a Splunk component that handles search requests and distributes them to indexers, but it does not process incoming data.

A heavy forwarder is a Splunk component that can perform parsing, filtering, routing, and aggregation on the data before forwarding it to indexers or other destinations. A heavy forwarder can break a stream of syslog inputs into individual events based on the line breaker and should linemerge settings in the inputs.conf file1.

An indexer is a Splunk component that stores and indexes data, making it searchable. An indexer can also break a stream of syslog inputs into individual events based on the props.conf file settings, such as TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD, and line_breaker2.

A Splunk component is a software process that performs a specific function in a Splunk deployment, such as data collection, data processing, data storage, data search, or data visualization.

Syslog is a standard protocol for logging messages from network devices, such as routers, switches, firewalls, or servers. Syslog messages are typically sent over UDP or TCP to a central syslog server or a Splunk instance.

Breaking a stream of syslog inputs into individual events means separating the data into discrete records that can be indexed and searched by Splunk. Each event should have a timestamp, a host, a source, and a sourcetype, which are the default fields that Splunk assigns to the data.

https://docs.splunk.com/Documentation/Splunk/8.0.4/Admin/Propsconf#props.conf.spec

https://docs.splunk.com/Documentation/Splunk/8.1.1/Admin/Propsconf

"* Reuse of the same field-extracting regular expression across multiple sources, source types, or hosts."https://docs.splunk.com/Documentation/Splunk/8.0.4/Admin/Propsconf#props.conf.spec

https://docs.splunk.com/Splexicon:Searchpeer

"A Splunk platform instance that responses to search requests from a search head. The term "Search peer" is usually synonymous with the indexer role in a distributed search topology..."

https://docs.splunk.com/Documentation/Splunk/8.1.3/Security/ConfigureLDAPwithSplunkWeb

"You can map either users or groups, but not both. If you are using groups, all users must be members of an appropriate group. Groups inherit capabilities form the highest level role they're a member of." "If your LDAP environment does not have group entries, you can treat each user as its own group."

In Splunk Enterprise, when collecting data from remote Windows machines using Windows Management Instrumentation (WMI), the configurations are defined in the wmi.conf file. This file specifies the parameters for connecting to WMI providers and defines the data inputs.

The wmi.conf file is located in the $SPLUNK_HOME\etc\system\local\ directory. It contains stanzas that define global settings and input-specific configurations for WMI data collection. This setup allows Splunk to collect various types of data from remote Windows systems, such as event logs and performance metrics, without requiring a forwarder on the remote machine.

https://docs.splunk.com/Documentation/Splunk/8.0.5/Updating/Updateconfigurations First line says it all: "The deployment server distributes deployment apps to clients."

When creating new user roles in Splunk, it's recommended to define roles that incorporate the necessary capabilities and index access permissions. This approach allows for granular control over user access and aligns with best practices for role-based access control.

"You can create custom roles and assign those roles to your users. Custom roles let you make granular adjustments to user access, including... Role inheritance... Capabilities... Allowed and default indexes..."

— About configuring role-based user access -

By creating roles that incorporate both capabilities and index access, administrators can efficiently manage user permissions and maintain a secure Splunk environment.

https://docs.splunk.com/Documentation/Splunk/8.1.1/Indexer/HowSplunkstoresindexes

Once further conditions are met (for example, the index reaches some maximum number of warm buckets), the indexer begins to roll the warm buckets to cold, based on their age. It always selects the oldest warm bucket to roll to cold. Buckets continue to roll to cold as they age in this manner. Cold buckets reside in a different location from hot and warm buckets. You can configure the location so that cold buckets reside on cheaper storage.

The correct answer is D. Place both props . conf and transforms . conf on the Heavy Forwarder for source A, and place both props . conf and transforms . conf on the indexer for source B.

According to the Splunk documentation1, to mask sensitive data from raw events, you need to use the SEDCMD attribute in the props.conf file and the REGEX attribute in the transforms.conf file. The SEDCMD attribute applies a sed expression to the raw data before indexing, while the REGEX attribute defines a regular expression to match the data to bemasked.You need to place these files on the Splunk instance that parses the data, which isusually the indexer or the heavy forwarder2. The universal forwarder does not parse the data, so it does not need these files.

For source A, the data is routed through a heavy forwarder, which can parse the data before sending it to the indexer. Therefore, you need to place both props.conf and transforms.conf on the heavy forwarder for source A, so that the masking takes place before indexing.

For source B, the data is routed directly to the indexer, which parses and indexes the data. Therefore, you need to place both props.conf and transforms.conf on the indexer for source B, so that the masking takes place before indexing.

In Splunk Enterprise, when integrating with an LDAP (Lightweight Directory Access Protocol) directory for authentication, user access is governed by the mapping between LDAP groups and Splunk roles. A user authenticated via LDAP must belong to at least one LDAP group that is mapped to a Splunk role. Without this mapping, the user can authenticate successfully against LDAP but will not be granted any role privileges inside Splunk, and therefore cannot log in to the Splunk web interface.

Splunk documentation explicitly states:

“When you integrate Splunk Enterprise with LDAP, a user must be assigned at least one Splunk role through an LDAP group mapping. If the user does not belong to a mapped group, they cannot log into Splunk.”

This ensures that user permissions are inherited from LDAP-to-role mappings and provides centralized management of authentication and authorization.

Reference (Splunk Documentation):

Splunk® Enterprise Admin Manual → Securing Splunk Enterprise → Authenticate users with LDAP

authentication.conf.spec and example → LDAP configuration and role mapping

Splunk Docs: “Configure LDAP authentication”

"The forwarder/indexer relationship can be considered platform agnostic (within the sphere of supported platforms) because they exchange their data handshake (and the data, if you wish) over TCP.

https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Typesofforwarders

"A heavy forwarder parses data before forwarding it and can route data based on criteria such as source or type of event."

https://docs.splunk.com/Documentation/Splunk/8.2.2/Knowledge/Usedefaultfields splunk_server

The splunk server field contains the name of the Splunk server containing the event. Useful in a distributed Splunk environment. Example: Restrict a search to the main index on a server named remote. splunk_server=remote index=main 404

When Splunk is integrated with LDAP, most of the user attributes are managed by the LDAP server and cannot be changed in the Splunk UI. However, one exception is the default app attribute, which specifies which app a user sees when they log in to Splunk. This attribute can be changed in the Splunk UI by editing the user settings. Therefore, option A is the correct answer. References: Splunk Enterprise Certified Admin | Splunk, [Configure Splunk to use LDAP and map groups - Splunk Documentation]

The coldPath parameter defines the path for the cold buckets, which are the oldest and least frequently accessed data in an index1. By setting the coldPath to point to the NAS mount point, the Splunk administrator can achieve the retention strategy of having older data on slower NAS storage.