PSP-Mal: Evading Malware Detection via Prioritized Experience-based Reinforcement Learning with Shapley Prior.
Dazhi ZhanWei BaiXin LiuYue HuLei ZhangShize GuoZhisong PanPublished in: ACSAC (2023)
Keyphrases
- malware detection
- reinforcement learning
- malicious executables
- anomaly detection
- state space
- application programming interface
- prior knowledge
- function approximation
- model free
- multi agent
- game theory
- learning algorithm
- reinforcement learning algorithms
- temporal difference
- machine learning
- markov decision processes
- optimal policy
- dynamic programming
- feature extraction