Learning Graph Representation of Agent Diffusers