I am inflating a view on button click and the user can add as many views as he likes, all is fine I made it work, but now the problem is when I go back one activity and come
As your application may be killed completely at any moment without noticem you have to provide long term storage off heap memory
You only have to restore all the views, if your activity was terminated (and it can be at any time). When it is activated again after termination, it goes through onCreate() method - this would be proper place to restore activity state.
Only callback which is guaranted to be called before your application / activity is destroyed is onPause() - this is a proper place to save views states into long term off-heap storage.