Discovering a Russian Cyber Intrusion

Geoblocking is a valuable tool in cybersecurity, but it has limitations. While many companies enforce outright bans on foreign traffic, a more nuanced approach is often required. In our case, the web application we manage is intended for U.S.-based users only, but that doesn't mean legitimate users won't occasionally access it from abroad. Business travel, remote work, and global VPN routing can all trigger foreign access.

Rather than hard-blocking all non-U.S. traffic, we use a tiered security approach:

• Public-facing pages (such as the homepage) are monitored for abuse patterns and probing attempts rather than geolocation alerts, reducing unnecessary noise.
• Authorized portal requests (requests made past the login wall) are flagged when originating outside the U.S. for further investigation.
• Certain high-risk regions, such as Russia, are hard-blocked outright. These requests are denied at the network level, but logs still capture the attempted access.

It was within this authorized-side alert system that we detected something unusual.

A request from Russia attempted to access specific private resources. Pages that should have been invisible to anyone without valid login credentials. This was immediately suspicious. Unlike the typical automated scanning we see, this attacker was not blindly guessing at endpoints.

They knew exactly what to look for.

The question was: How did they know those private resources existed?

Identifying the Breach: Tracing the Source

With the unusual Russian requests in our logs, the next step was clear: find out how they knew these private URLs existed.

Step 1: Searching for a Pattern

We queried our logs for all requests to the affected entities. The goal was to identify if any legitimate users had accessed the same pages prior to the Russian requests.

A pattern quickly emerged.

One specific user from North Carolina had accessed the exact same entities earlier that day. Every entity the Russian actor attempted to access had already been viewed by this user approximately eight hours prior.

Step 2: Analyzing the Requests

The similarities between the requests were striking:

• The URLs matched exactly, including entity IDs.
• The order of the requests was identical, down to the sequence in which the user navigated.
• The attack requests lacked authentication headers, meaning they had the structure but not the credentials to retrieve actual data.

This was not a typical brute-force attack. The attacker was not randomly guessing entity IDs. Instead, they had a precise roadmap of what to request.

Step 3: Ruling Out Other Explanations

We considered every reasonable alternative before concluding the user's system was compromised:

• A Man-in-the-Middle Attack? Unlikely. The requests were encrypted and the attacker could not decrypt them.
• A Shared Link Leak? Highly improbable. In this web application, users do not typically share internal links. Even if they had, the access pattern would vary.
• A Third-Party Integration Breach? No external services were involved in this use case.
• VPN Spoofing? Also unlikely. Why would a legitimate U.S.-based company VPN into Russia? Additionally, the requests were out of business hours in North Carolina.
• A Supply-Chain Attack? No evidence of compromised dependencies in the user's flow.

Every alternative explanation was ruled out. The only remaining possibility? The user's system or network was compromised, and something was harvesting their browsing data.

Step 4: Confirming the Timeline

The final clue that sealed our theory was timing.

• The user accessed the entities during their normal workday in North Carolina.
• The Russian actor attempted the same requests eight hours later, after the North Carolina office's business hours.

This confirmed that the attacker was not acting independently. They were reacting to real user activity in near real-time.

With this information in hand, we needed to act quickly.

Taking Action: Containing the Threat

Once it became clear that a compromised system was leaking internal activity to an external attacker, immediate action was required.

Step 1: Locking the User Out

Security is not something we take lightly. The moment we confirmed the breach pattern, we immediately locked out the affected user. This was done without hesitation and without waiting for client approval.

This was a non-negotiable security decision and until we had a clear answer, the compromised user was a liability. While this decision could have caused friction, we would rather lose a contract than knowingly allow an insecure system to persist.

Once the lockout was in place, we informed our direct client of the situation using SMS and Slack. For this client, it was the fastest available communication channels. Since they responded right away, no email follow-up was necessary at that stage.

Step 2: Notifying the Affected Office

Because the breach originated from a client of our client, we worked with our direct client to inform them.

• With their approval, we drafted a technical email explaining our findings and CCed our client.
• Our client followed up with a phone call to emphasize the seriousness of the issue.
• The North Carolina office's leadership, however, was not particularly technical and initially dismissed the possibility that they had been compromised.

Step 3: The Initial Pushback

The North Carolina office's first reaction was skepticism. They insisted that a security breach on their network was impossible since they had adequate protections in place. A "it could not happen here" mentality.

However, with their account locked out of a critical system, they had no choice but to escalate the issue to their internal IT team.

Step 4: Confirmation of the Infection

A few days later, we received an update:

• Their IT team did find malware on their network.
• They opted for a full system wipe rather than trying to remove individual infections.
• Since this was a small office, the process was relatively quick.

Unfortunately, since we only communicated with their leadership, we were never given the exact details of the malware type or infection method. In hindsight, pushing for this information might have been beneficial, but the office seemed eager to move on from the incident.

Step 5: Verifying the Resolution

The final confirmation came when we monitored the logs after their cleanup and restoring their access at a limited level.

• After their systems were wiped, the Russian requests stopped completely.
• No further unauthorized attempts were detected.
• This strongly confirmed our original theory that the breach was entirely tied to their compromised network.

With the issue resolved, we restored full access to the affected user.

Conclusion: Complacency and the Cost of Overlooking Anomalies

Security breaches do not always begin with a catastrophic failure. More often, they start with a small anomaly. Something that seems insignificant but is actually the first in a chain of failures leading to a serious incident.

The Russian requests we discovered were, on their own, benign. They lacked authentication and were ultimately blocked. It would have been easy to dismiss them as just another meaningless attempt from an automated scanner. But security is not just about responding to active threats, it is about recognizing early warning signs.

Leadership's Role in Security

This incident reinforced a critical reality for business leaders:

1. Security threats are not always obvious.
   • Attackers do not always break in through the front door. Sometimes, the first sign of a breach is indirect and subtle; a log entry, an unusual access pattern, or a misalignment in expected behavior.
2. Ignoring weak signals today can lead to disasters tomorrow.
   • In aviation, we often talk about redundancies. There multiple layers of defense designed to prevent catastrophic failure.
   • Accidents do not usually happen because of a single issue, they happen because a series of failures compound.
   • The first failure was the North Carolina office's security being breached. If we had ignored this anomaly, we would have been the second failure in that chain. How many failures did our client and their client have left before something truly damaging occurred?
3. Trust your IT teams when they tell you something is wrong.
   • When leadership dismisses security concerns, they are not just ignoring risks... they are actively allowing them to grow.
   • IT professionals often detect signals that seem too small to be a concern but are actually early indicators of something larger.
   • The difference between catching an issue early and facing a crisis later often comes down to whether leadership is willing to listen.

A Call to Action: Standing by What Is Right

For those managing critical systems, this incident serves as a reminder:

• Never let complacency set in. Just because you have seen an alert before does not mean the next one is harmless.
• Stand by what you know is right. Taking action, even when it causes friction, is part of responsible system administration.
• For leadership: Take security seriously. If your IT team tells you something is wrong, do not ignore them. You might not see the danger, but they do.

Security is not just about technology, it is about mindset. The difference between preventing a breach and dealing with its consequences often comes down to whether small anomalies are recognized for what they truly are: early warnings.