Solmio-kassa - Elavon varmennuksissa häiriöitä – Incident details

Tidypayn Euroopan maksuvälityspalvelussa ilmeni 2.–3.8.2025 häiriöitä, jotka aiheuttivat hitautta ja virheitä korttimaksuissa. Häiriö liittyi ulkoisen palvelun istuntohallintaan, joka ylikuormittui käsitellessään suuria korttitietojen tokenointieriä.

Tilanne on nyt vakautettu, ja kaikki tapahtumat on käsitelty onnistuneesti.
Joillain asiakkailla viikonlopun maksujen tilitys voi viivästyä yhdellä arkipäivällä.

Tilapäisesti maksujen käsittelyä ohjataan manuaalisesti.

Incident report & RCA

Tidypay AS – Incident Report

Incident ID

TP-2025-07-02-TP-VaultSession

Date

Saturday 2 – Sunday 3 August 2025

2 August Start to End (CEST)

17:50 to 18:30 CEST, (slow declined transactions)

3 August Start to End (CEST)

13:05 to 15:15 CEST (intermittent slowdowns)

Detected by

Automated latency alerts & Customer reports

Severity

High – intermittent payment‑token service degradation Europe region

Affected components

Tidypay gateway external token vault

Customer impact

Slower transaction processing; intermittent tokenisation failures for Elavon batch merchants

Summary

Between 17:50 CEST on 2 August and 15:00 CEST on 3 August 2025, Tidypay’s European gateway experienced elevated latency and intermittent failures when processing large batch tokenisation jobs routed through its new secure‑token vault. About 40 minutes of severe degradation (complete batch processing halt) were observed on Saturday, followed by 70 minutes of recurring slowdowns on Sunday until the underlying issue was identified and mitigated.

Timeline (CEST)

Time (CEST)

Event

17:50 2 August

Automated latency alerts and customer reports of transaction slowdowns as an unfinished Elavon batch job was automatically restarted.

18:00 2 August

Incident bridge opened; full Tidypay team assembled.

18:30 2 August

Batch job manually terminated; traffic restored to normal parameters (root cause still unknown).

13:20 3 August

Monitoring detects increased average transaction latency; customer complaints resume.

13:35 3 August

Emergency bridge re‑opened; external partners invited.

13:50 3 August

External partners joined bridge; live log analysis began.

15:00 3 August

Root cause confirmed: Token‑vault session‑exhaustion due to unexpired TLS sessions per request.

15:15 3 August

Work‑around applied: large batches split into ≤10 k records each.

Root Cause

Tidypay’s recently deployed integration towards an external provider introduced a new, dedicated TLS session per tokenisation request. External token provider configuration retained each session in memory for 120 minutes. During high‑volume batch imports (~45 000 requests in < 2 h) the vault exhausted available memory, ceasing to respond and causing the gateway to queue or fail subsequent requests. The issue recurred whenever a large batch was processed, explaining the Saturday and Sunday degradations.

Mitigation & Resolution

• Immediate: offending batch terminated and rerun in 30k chunks, preventing further session exhaustion. • External token vault: configuration change requested to reduce idle‑session TTL from 120 minutes to 5 minutes. • Tidypay: gateway hot‑patch to reuse persistent connections and throttle batch concurrency.

Corrective & Preventive Actions

1. Short‑term (August 2025): automate batch‑splitting in gateway; alert on active session count. 2. Medium‑term (Q3 2025): deploy dedicated in‑datacentre token vault with configurable session limits; evaluate alternate providers. 3. Long‑term (Q4 2025): implement multi‑region vault fail‑over and load‑shedding on tokenisation path.

Customer Communication

Affected merchants were notified via email from 18:15 CEST on 2 August and from 13:37 CEST on 3 August with an incident summary and follow up confirmation that all impacted transactions were re‑processed successfully. Some merchants might experience a 1 day delay in the settlement for transactions processed during the weekend.

We implemented a fix and are currently monitoring the result.

Transactions getting through now