Click trap: not only for users, but also for link analyzers

Click trap: not only for users, but also for link analyzers 🐭

When manually analyzing links, we typically ask ourselves only one question — “is it safe?” ✔️❌

Transferring a similar approach into a streaming environment for analysis is not possible: actions initiated by following a link can lead to new problems. For example, invitations from email messages may be automatically accepted or declined, automatic unsubscribing and/or subscribing to correspondence may occur, and so on. If such an action causes the service to send a new email with similar links, then automatically following them will cause an incessant “avalanche” of messages, which can lead to reduced performance of security solutions or simply to user annoyance.

🫵 To solve this problem, an approach can be proposed that is logically similar to the process of selecting links during manual analysis: some links seem extremely suspicious to us or clearly require additional context for analysis, while others are definitely safe or, as in the previous situation, trigger certain actions on services (or are generally one-time). An approach where the system defines a set of indicators for “definitely need to follow” and a set of indicators for “definitely do not need to follow” is called rule-based decision-making (rule-based decision system). What indicators can be proposed for implementing such a system?

In the set of conditions preventing following a link, the following can be considered:

• the presence of patterns in the URL path that semantically indicate a possible action upon activation, for example, /(un)subscribe/, /login/, /exit/, /action/, /track/, etc.;

• the presence of query parameters token, key, uid with values whose format matches UUID or JWT;

• the presence of query parameters ts or expires indicating the link’s lifetime;

• if the link came from an email message, its presence can be checked in separate subscription/unsubscription headers — List-(Un)Subscribe:;

• if data streams are provided with a set of “white” domains, disabling analysis of URL links containing them can be considered. This option should be used more carefully, since even the most popular and well-known services and domains can be used in scenarios with content redirection.

In the set of conditions causing a link to be followed, the following can be considered:

• links with an explicit IP address and/or non-standard port;

• links with a recently registered domain;

• if a web resource categorizer with such classification is available — links to shortener services. If not available, a condition based on short URL length can be considered;

• links with files in the URL path that have specific static extensions, for example, *.pdf, *.exe, etc.;

• links to object storage services, for example S3 or IPFS storage.

🧐 What should be done with links from the analyzed object that did not fall under any of the lists? Here one can rely on the real picture that emerges after such an algorithm runs: if system performance allows or the analysis time does not exceed the acceptable SLA for processing objects, all “gray” links can be sent for content retrieval. Or a limit can be introduced on the number of simultaneously analyzed links for a single object.

Also, for URL links that were not categorized by the decision-making system, a “cautious” algorithm for obtaining additional context for analysis can be provided: follow the link using the HEAD method without retrieving content. In this way, additional information can be obtained from HTTP headers, applicable both within the algorithm described above and generally for deciding whether a link is malicious.

#tip #url #mail
@ptescalator
💬 X 💬 Max