Can we use regular expressions in the search?

question

#1

I was trying to do a phrase search to find some keywords in the $LogEvent field. While doing this I tried putting up regular expressions between the single quotation marks. This was yielding results but they were incorrect.

for an Example, if in web access logs, if I want to search for 400 status code, there are two different occurrences of the term ‘400’ one is the actual status code, the other being some random string. In such a case, I used the below query to pull only the logs of my choice, which is completing successfully but giving wrong output.

_fetch * from event where $DevSrcIP=A.B.C.D AND $LogEvent='400\s\"CONNECT\s' limit 10

My question is, is it valid to use regular expressions in the search query? if no, what is happening to my search when I am using the above search. and if we can use regex in DQL, how do I use it?

P.S. The above is just an example, and I am aware, I can get it done by querying parsed fields.


#2

Hi Ashutosh, The LogEvent field is the raw log and this field is analysed field where all words are being treated as a separate string.
So when you search something in LogEvent field any occurrence of that search in any log will be shown in the stack and fuzzy search supports wildcard characters like * to search data but if you have special characters in the search string it may not work as expected.
You can use regex function only in non-analysed field not in LogEvent Field or you can use fuzzy search in LogEvent field.
If you have DNIF 8.1.1 and above you can use regex in _fetch directive on non-analysed fields.
Use $FieldName=regex(yourregex)


#3

Thanks Ravi. This provides me a better picture.