Abstract

Pattern-based malware detection has been prevailing in the security industry for years. The traditional workflow usually takes a lot of time to collect and analyze new samples.
How to automatically cluster samples?
How to automatically evaluate the quality and value of each cluster?
How to automatically verify the quality of patterns extracted from one cluster?
These are common problems researchers are dealing with on hundreds of thousands of samples every day.

In this topic, we propose a new system, DiDe, to address all of these problems. By consuming streaming data from VirusTotal, this system can cluster samples with high quality at scale. After a signature is generated, DiDe keeps monitoring its performance, by comparing the detections of the signature and the cluster the signature is generated from, it can keep evaluating and updating the quality of the signature. In addition, this topic also discusses the different ways of sample clustering and why we chose label aggregation.