I co-authored a blog post with my lab that has practical advice for debugging DNNs - some of these tips might be helpful to you? https://pcc.cs.byu.edu/2017/10/02/practical-advice-for-build...