Instrument 04 Find duplicate logs across streams

Find duplicate Loki logs across streams

For Loki operators checking whether duplicate ingestion inflates alerts and storage.

Try it with sample data See two duplicate groups now. Install the CLI

Processes samples in this browser
Browser input limit: 5 MB
Evidence, not verdicts

Cutaway illustration of three log signal traces converging through a lens into one paper-chart pulse — Three stream traces through one suspected duplicate group

01Read messagesHide changing IDs
02Set a retry windowMatch retry timing
03Compare streamsShow differing labels
04Inspect groupsReview duplicate evidence

Bench test · browser edition

Check a log sample

Paste JSON lines, a Loki response, or plain lines. The browser reads up to 5 MB.

Input ALocal circuit

Log sample

Redact sensitive values before sharing the exported report.

Choose file

Retry window

0.5 s 2 s 5 s

Readout BAwaiting sample

No sample on the bench

Open the sample analysis or choose a log export.

A match is a lead, not a verdict.

How to assess a suspected duplicate

What the readout can—and cannot—tell you

Evidence for a duplicate

Look for one normalized message across streams within the retry window.

Cases that need context

Health checks and expected fan-out can look alike. Review the stream labels.

Check retry and sharding settings

Check retry intervals, sharding labels, and producer request IDs before changing a pipeline.

Single binary

Put the lens beside your logs

Run the bundled sample before pointing the tool at a log export.

LOCAL / OPERATORUTF-8

cargo install --git https://github.com/B-Divyesh/sf-log-duplicate-lens
log-duplicate-lens demo

Demo: uses the bundled seven-record sample and writes a temporary report.

JSON: use --json for a machine-readable report.

Limits: set --max-events, --max-groups, and --max-input-mb.

CLI DEMO / BUNDLED SAMPLELOCAL

$ log-duplicate-lens demo
Bundled sample: 7 labeled log records
Result: 2 suspected duplicate groups / 3 duplicate copies
Demo report: /tmp/log-duplicate-lens-demo-….json