Hold Shift and then press Delete. (This forces Windows to skip the recycle bin as the cache can be quite large and is comprised of temporary files.)
