# [[Structure of LogQL]] ![[Structure of LogQL.svg]] Queries run using [[Grafana Loki|Loki]]'s [[LogQL]] are structured in this way: ``` { log stream selector } | log pipeline ``` ### Log stream selector ``` {service_name="nginx", status="500"} ``` The log stream selector is a string containing key-value pairs. The unique combination of all these pairs is called a stream. The goal of the log stream selector is to narrow down the data set to be processed and filtered in the log pipeline by using either predetermined (in the Loki config) or automatically detected labels. The following operators can be used in the log stream selector: - = - != - =˜ - !˜ where the `˜` character is used for [[Regular Expressions]]. Here's an example of a regular expression used within the log stream selector: ``` {name =~ "[\S\s]*mysql[\S\s]*} ``` This matches log label values with newline character. > [!tip] Default label > `service_name` is a default label that Loki creates and tries to populate with something in the log line that looks like a service name. This is to enforce the good practice of having standardization in log labels. However, the default can be changed in Loki configuration. ### Log pipeline The log pipline is optional and may contain any combination of the following expressions: - filters - parsers - formatters #### Filter expressions Filter expressions are used to look for a certain string or range of values within the log line or label. For line filters, the following operators can be used: - `|=`: Log line contains string - `!=`: Log line does not contain string - `|~`: Log line contains a match to the regular expression - `!~`: Log line does not contain a match to the regular expression For label filters, the following operators can be used: - == or = for equality. - `!=` for inequality. - `>` and `>=` for greater than and greater than or equal. - `<` and `<=` for lesser than and lesser than or equal. `and` and `or` can also be used to chain operations for label filters. #### Parser expressions Parser expressions are used to easily filter logs that are in a standardized format, such as: - [[JSON]] - logfmt - [[Regular Expressions]] - unpack In this example, the `json` parser is used to get the parameters `request_method` and `request_uri` and use them to filter log lines: ``` {$label_name=~"$label_value", job=~"$job", instance=~"$instance"} | json | line_format "➡️ {{.request_method}} {{.request_uri}} with HTTP status: {{.status}} " ``` [^ward] #### Format expressions Format expressions can be used to change the way lines (`line_format()`) and labels (`label_format()`) are displayed. Note that format expressions do not change the underlying source data, only the results that are returned by the query. The example above uses a `line_format()` function to rewrite the original log line: ``` {"msec": "1733828027.704", "connection": "81290341", "connection_requests": "1", "pid": "24", "request_id": "7a98025445386b1feef1402bb13362e5", "request_length": "559", "remote_addr": "52.167.144.255", "remote_user": "", "remote_port": "", "time_local": "10/Dec/2024:10:53:47 +0000", "time_iso8601": "2024-12-10T10:53:47+00:00", "request": "GET /a/1080936380/alternative-to-refaid-refugee-aid-app.html HTTP/1.1", "request_uri": "/a/1080936380/alternative-to-refaid-refugee-aid-app.html", "args": "", "status": "200", "body_bytes_sent": "11497", "bytes_sent": "11676", "http_referer": "", "http_user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36", "http_x_forwarded_for": "52.167.144.255", "http_host": "nl.appfelstrudel.com", "server_name": "ns565366.ip-54-39-133.net", "request_time": "0.105", "upstream": "172.19.0.255:3006", "upstream_connect_time": "0.000", "upstream_header_time": "0.100", "upstream_response_time": "0.100", "upstream_response_length": "72787", "upstream_cache_status": "MISS", "ssl_protocol": "", "ssl_cipher": "", "scheme": "http", "request_method": "GET", "server_protocol": "HTTP/1.1", "pipe": ".", "gzip_ratio": "6.34", "http_cf_ray": "8efcb173cc760648-IAD","geoip_country_code": "US"} ``` to this much more readable line in the results: ``` ➡️ GET /a/1413644634/alternative-to-balaji-quotes-hindi.html with HTTP status: 200 ``` > [!warning]- Avoid label formats for metrics > Doing `label_format()` for metric queries is very expensive. We recommend using [transformations within Grafana](https://grafana.com/docs/grafana/latest/panels-visualizations/query-transform-data/transform-data/) on the visualization side instead. %% # Excalidraw Data ## Text Elements ## Drawing ```json { "type": "excalidraw", "version": 2, "source": "https://github.com/zsviczian/obsidian-excalidraw-plugin/releases/tag/2.1.4", "elements": [ { "id": "4y8R7iOA", "type": "text", "x": 118.49495565891266, "y": -333.44393157958984, "width": 3.8599853515625, "height": 24, "angle": 0, "strokeColor": "#1e1e1e", "backgroundColor": "transparent", "fillStyle": "solid", "strokeWidth": 2, "strokeStyle": "solid", "roughness": 1, "opacity": 100, "groupIds": [], "frameId": null, "roundness": null, "seed": 967149026, "version": 2, "versionNonce": 939059582, "isDeleted": true, "boundElements": null, "updated": 1713723615080, "link": null, "locked": false, "text": "", "rawText": "", "fontSize": 20, "fontFamily": 4, "textAlign": "left", "verticalAlign": "top", "containerId": null, "originalText": "", "lineHeight": 1.2 } ], "appState": { "theme": "dark", "viewBackgroundColor": "#ffffff", "currentItemStrokeColor": "#1e1e1e", "currentItemBackgroundColor": "transparent", "currentItemFillStyle": "solid", "currentItemStrokeWidth": 2, "currentItemStrokeStyle": "solid", "currentItemRoughness": 1, "currentItemOpacity": 100, "currentItemFontFamily": 4, "currentItemFontSize": 20, "currentItemTextAlign": "left", "currentItemStartArrowhead": null, "currentItemEndArrowhead": "arrow", "scrollX": 583.2388916015625, "scrollY": 573.6323852539062, "zoom": { "value": 1 }, "currentItemRoundness": "round", "gridSize": null, "gridColor": { "Bold": "#C9C9C9FF", "Regular": "#EDEDEDFF" }, "currentStrokeOptions": null, "previousGridSize": null, "frameRendering": { "enabled": true, "clip": true, "name": true, "outline": true } }, "files": {} } ``` %%